System and Method for an End-Device Modulation Based on a Hybrid Trigger

ABSTRACT

A system for end-device modulation by a hybrid trigger comprising at least one end-device (E-D) in communication with at least a first device (D1) outputting audio/video (a/v) programming; a processor; a memory element coupled to the processor; a program executable by the processor to: position a virtual representation of the E-D on a digital canvas displayed on a D1-coupled display representing a user&#39;s physical and virtual space; and capture from a corresponding region of the virtual space for modulating an effect on a corresponding portion of the canvas-positioned E-D based on a combination of at least two different triggers recognizing at least one of an a/v element or a/v event from the a/v program.

BACKGROUND Field

The field of the invention relates to sensory delivery systems, and moreparticularly, relates to a precise automated haptic system withprogrammable logic for the latent-free and target specific delivery ofvariable air flow and temperature to mimic a realistic somatosensoryexperience in an immersive entertainment environment. More specifically,the invention relates to actuating any number of peripheral devicesbased on an unscripted feed using computer vision logic. Even morespecifically, the invention relates to delivering a customizableimmersive experience from peripheral devices based on end-user scripted(EUS) effects on a rendered web-browser page.

Related Art

Virtual Reality (VR) aims to simulate a user's physical presence in avirtual environment. Over the past decade, with the rapid development ofcomputer-generated graphics, graphics hardware, and modularization ofprocessing elements and system components, VR has been ushered into thenext revolution—Immersive Multimedia. Small-form factor devices, such asdata gloves, haptic wearables, and head-mounted gear, have all enhancedthe immersive experience in the virtual reality environment. Now, withthe advent of sophisticated tracking technology, this immersiveexperience has even extended to the cinema experience; viewers will beable to change their perspective on a scene based on the positiontracking of their eye, head, or body. This immersive and active viewingexperience is poised to alter the way in which we will consume contentin the future.

Along with a number of immersive developments in the virtual realityindustry, there have been a number of developments in enhancing thesensory experience for a user. For example, force feedback in medical,gaming, and military technology is very well known in the art. 4-D movietheaters, replete with motion rocking, have long been providing viewerswith a life-like experience. Developers have increased the sensorydefinition by stimulating a plurality of senses with an exceptionallyhigh degree of realism.

Scientists from York and Warwick in England have developed a virtualreality cage called a Virtual Cocoon, in which a user is enveloped by aplanetarium-style screen, not only surrounded by a stereoscopic visualand sound, but also by a sense of smell, touch, and even taste. Thisfully immersive, perceptual experience blurs the line between what isreal and what is not. Holovis manufactures full motion domes—immersiveand interactive platforms designed primarily for gaming, but can bescaled up for group interactive experiences. Stereoscopic projectors areedge blended and synchronized with ride motion technology, along withdelivering a range of other sensory stimulants, such as smell and heat.

Likewise, there are a number of patent references providing for VRsystems that deliver haptics. However, much like the Cocoon and Holovis,the background patent references provide a plurality of sensorymechanisms integrated with a user-surrounding platform or rig. The useof VR or entertainment platforms featuring a plurality of sensorymechanisms is well established in the background art, but not asindividualized devices with home-use and universal integrationcapabilities. Moreover, there are no claims or disclosure in the priorart addressing individualized units coupled to a code instructingvariable air intensity and temperature, stimulating a wide range ofvariable haptic situations in a virtual reality environment.

What's more, none of the extant systems teach a system or method forprocessing the audio/video input for generating a real-time hapticcommand output, wherein the said output drives a variety of hapticeffects from the modular haptic tower: wind effects, velocity, suddenimpact, blast, water misting, and, or strike impact or pressure. As theforegoing illustrates, there is currently a gaping void for a home-use,stand—alone device, that may integrate into a variety of experiencesystems, and deliver target specific haptics with next generationrealism and with virtually zero latency. Users no longer will have torely on attending a VR convention or gaming room in order to experiencethis heightened immersion and sensory experience. No longer will theyhave to commit to large and cumbersome installations and platforms.Finally, with targeted haptics delivery, the sense of realism andimmersion will be taken to the next level—all from the convenience ofone's own home, and most importantly, free from content support hurdlestrapping content within provider and developer silos.

Extant systems do not employ learning based approaches to complement theuser input or virtual environmental input in order to provide additionalcontext for a haptic command. Extant systems do not continuously learnand update a deep neural network or discriminative library, whichattempts to dynamically learn the haptic-commanding events in a user'ssurrounding, in order to create shortcuts in the input processing. Suchshortcuts may cut down on latency between input and haptic output,providing for a substantially more real-time experience. Moreover, suchshortcuts may reduce the load bearing of the system and increase overallcompute efficiencies. Learning based approaches may additionally predictfor location of an event at time interval t, and furthermore, predict avariety of coefficients based on a reference parameter, and command fora specific haptic output. However, extant solutions for reactively andpredictively tracking events in a virtual environment are lacking, andtherefore, there is a need for a computationally efficient solution forsolving the problem of event tracking (reactively and predictively) in avirtual environment, and coupling to a haptic command/output withvirtually no latency.

Nothing in the prior art teaches for directly integrating a peripheraldevice to audio or video signals from an original programming feed or alive feed to trigger or control at least one of actuation or hapticeffect based on computer vision processing of said audio or videosignals. In other words, the actuation or haptic effect is not triggeredby embedding triggering cues via a developer kit or after-market coding(scripted programming feed), but rather, directly integrative to theoriginal programming feed or live feed in a plug-n-play fashion viacomputer vision processing (unscripted programming feed)—therebyobviating content hurdles and opening the full library of a/v basedprogramming in communication with a peripheral device, whether it be aendoscope, security surveillance, television show, video clip, audioclip, social media integration, electronic communications featuringaudio/video/emojis, movie, sporting event, gaming, virtual environment,augmented environment, real environment, etc. Examples of peripheraldevices may be any device capable of an actuation or haptic effect andmay be in contact with a user or free from a user, such as, watches,gloves, wrist bracelets, pants, shoes, socks, head gear, wearables,sleeves, vests, jackets, heat lamps, haptic towers, light fixtures,speakers, medical interventional tools, mobile phones, tablets, displayscreens, remote controllers, game controllers, 4-D movie theater seats,stadium seats, etc. Users may now finally be free from content supporthurdles trapping content within provider and developer silos and unlockthe fourth dimension of the immersive experience by simply plugging andplaying.

Nothing in the prior art teaches for offering customizable effects fromend-peripheral devices beyond standard user-interface controls fordelivering a more customized and immersive viewing or interactiveexperience. Moreover, the prior art does not teach for a basic,low-level script input by an end-user for customizing effects controlsbased on a user preference concomitantly with the viewing or interactiveexperience. Currently, effects driven on end peripherals requiresophisticated front-end programming (C+/C++) to code for mapping theeffects on end-peripherals; not providing a low-coding barrier optionfor end-users to script customized immersive effects and render thescripted web-page to effect-ripple the end-peripheral accordingly.

Finally, the prior art is silent on solutions for eliminating falsepositives and/or latency in the delivery of end-device modulation (interalia, lighting effects) during a gaming/viewing experience. Furthermore,the prior art is silent on providing solutions for combinatorial/layeredeffects as a result of a combination of scripted/unscripted triggersfrom an audio/video output corresponding to the viewing/gamingexperience. Eliminating false positives and latency, in addition toproviding combinatorial end-device effects, is critical in theenhancement of an immersive experience for a viewer/gamer—as they forgedeeper into the virtual realm in the wake of physical distancingmeasures due to COVID-19.

SUMMARY

These and other features and improvements of the present applicationwill become apparent to one of ordinary skill in the art upon review ofthe following detailed description when taken in conjunction with theseveral drawings and the appended claims. This invention relates to thenext generation of Immersion Multimedia, in which variable air flow andtemperature haptics delivery is targeted to specific portions of theuser corresponding to the user in the Virtual Space. Moreover, theapparatus, system, and method, do not rely on an installation orplatform, but rather, is modularized for universalized integration. Thepresent invention fills a void left behind by the currently existingImmersion Multimedia products and references. The present inventionprovides for an apparatus, system, and method for the precise haptictargeting of specific portions of a user—mimicking conditions of theVirtual Space—in a modularized, universally integratable form.

In one generalized aspect of the invention, the air haptic devicesimulates variably intense wind, heating and cooling from the virtualspace to enhance the user's sense of immersion. The hardware willinclude hot, cold and ambient settings with variable intensities for hotand cold based on power input and desired output temperature.

The apparatus may comprise a housing; at least one fan assembly; atleast one duct; at least one temperature element; a processor; a memoryelement coupled to the processor; encoded instructions; wherein theapparatus is further configured to: receive data input from a user;receive data input from a program coupled to an experience; based on thereceived input data, control an air flow intensity; based on thereceived input data, direct air flow through at least one duct; based onthe received input data, control a temperature element for heating orcooling the said air flow; and deliver a haptic output to a user.

In one preferred embodiment, the apparatus may be in the form of ahaptic tower that individually has the capability to blow air at hot andcool temperatures with variable intensity. The fan assembly will havethe capability to create a smooth, uniform flow of air, as opposed to anaxial-style fan, which “chops” the air, resulting in a non-uniform flowof air. In one preferred embodiment, a variable control of air flow maybe created by a variable controlled speed output from a motor actuatedfrom a series of sensor-captured and code-instructed data inputs. Inanother embodiment, a variable controlled electro mechanical valve canvary intensity of air flow and pressure. Some embodiments may includethe motor output to be coupled to a brake for tight control of thehaptic air flow.

In one aspect of the invention, air temperature may be created bycontrolling the redirected air flow through heat sinks of hot and cooltemperatures. Servo motors control dampers, flat plastic shutters, andthese shutters will open and close controlling the air flow throughdifferent temperature ducts. After redirecting the air into one of thethree separate ducts, each duct has either cold, hot or no temperaturetreatment to the out-flow of air. In this particular embodiment, the airflows through the “hot” duct with an exposed heating element. In someembodiments, for the hot duct, the air may flow through an exposedPositive Temperature Coefficient (PTC) ceramic heater element. In otherembodiments, the heating element may be a condenser heat sink in avapor-compression cycle, thermoelectric heating using Peltier plates,Ranque-Hilsch vortex tube, gas-fire burner, quartz heat lamps, or quartztungsten heating, without departing from the scope of the invention. Forthe “cold” duct, the air flows through a cooling element. In someaspects of the invention, for the cold duct, the air may flow through atraditional finned air conditioning evaporator in a vapor-compressioncycle. Alternate embodiments of the cooling element may includethermoelectric cooling using the Peltier effect, chilled water cooler,Ranque-Hilsch vortex tube, evaporative cooling, magnetic refrigeration,without departing from the scope of the invention. The last duct hasambient air bypassing both the heating and cooling elements. In anotheraspect of the invention, heating and cooling elements are integratedinto a single duct providing for heated air, cooled air, and ambientair. In yet another aspect of the invention, more than three ducts maybe provided in order to create heated air, cooled air, and ambient air.

It is a further object of the invention to provide an apparatus that mayhave an integrated air bursting element, delivering high velocity airflow directed at the user. In one embodiment, an array of miniaturespeakers may be used to create a large enough volume of air displacementwithin a chamber to generate a miniature air vortex. Another embodimentfor the air bursting effect may entail air displacement with the use ofa larger speaker or a sub-woofer. These are able to displace more air inan electromechanical fashion. Other embodiments may include air vorticesto create air bursting effects by attaching a rod supported by a railsystem powered by a motor assembly. In yet another embodiment, an aircompressor coupled to an electromechanical valve may be used to createthe air bursting effect.

In a preferred embodiment, target specificity for haptic delivery may beachieved using servo motors to pivot in place. In other embodiments,target specificity may be enhanced by using head tracking or full bodytracking sensors. In yet another embodiment, this body tracking can alsobe used for the control and aiming of the dispensing nozzle atparticular tracked body locations. An alternate embodiment may includenozzles that may shift the diameter of an outlet in order to alter theair flow pressure and haptic effect. The system may comprise aprocessor; a memory element coupled to the processor; encodedinstructions; at least one sensing means configured for detecting datarelated to a user's orientation and position, environmental conditionsin user's real environment, and user's input signal; wherein thecomputer system is further configured to: receive data input from auser; receive data input from a program coupled to an experience; basedon the received input data, control an air flow intensity; based on thereceived input data, direct the air flow through at least one duct;based on the received input data, control a temperature element forheating or cooling the air flow; and deliver a haptic output to a user.

In a preferred embodiment, a system configuration may comprise a modularsurround haptic system with multiple towers. The multiple towerconfiguration may have a micro controller controlling all of the towers.In some embodiments, communication between the micro controller and theCPU will be USB. Other embodiments may allow communication between themicro controller and CPU by other known methods in the art. In someembodiments, the towers will be in direct communication with the CPU viaany known communication protocol.

In one aspect of the invention, a system configuration may comprise asensor to detect data related to a user's orientation and position,environmental conditions in user's real environment, and users inputsignal. In another aspect of the invention, a user may be surrounded bya plurality of sensors to detect data related to a user's orientationand position, environmental conditions in user's real environment, andusers input signal. In other embodiments, the sensors may also includebody-tracking, hand-tracking, head-tracking, or eye-tracking technologyto be used for the control and aiming of the tower and nozzle atparticular track body locations in order to achieve high resolutiontarget specificity for haptic delivery. In further embodiments,sensor-captured data may communicate directly with the micro controller.In yet further embodiments, sensor-captured data may communicatedirectly with the towers, bypassing the micro controller.

It is yet a further object of the invention to provide a system andmethod that may comprise receiving data input from a user; receivingdata input from a virtual environment comprising the user; and said dataprocessed and converted for commanding control of any one of, orcombination of, an air flow intensity from a fan assembly and, or airdisplacement chamber; directing the air flow through at least one duct;controlling a temperature element for heating or cooling the air flow;controlling a water mist unit for wet effects; and, or controlling atactile member for delivering a strike or pressure impact to the user.

In yet another object of the invention, the system may be coupled to aneural network or machine learning approach, whereby the systemcontinuously learns and updates a deep neural network or discriminativelibrary. By doing so, the system may dynamically learn thehaptic-commanding events in a user's surrounding and create referenceparameters in order to create shortcuts in the input processing. Suchshortcuts may cut down on latency between input and haptic output,providing for a substantially more real-time experience. Moreover, suchshortcuts may reduce the load bearing of the system and increase overallcompute efficiencies. Learning based approaches may additionally predictfor location of an event at time interval t, and furthermore, predict avariety of coefficients based on a reference parameter, and command fora specific haptic output. Therefore, there is a need for acomputationally efficient solution for solving the problem of eventtracking (reactively and predictively) in a virtual environment, andcoupling the tracked event to a haptic command/output. Aspects andadvantages of this invention may be realized in other applications,aside from the intended application of gaming/interactive storytelling/cinema/passive story telling. Other pertinent applications thatmay exploit the aspects and advantages of this invention are:tourism—simulation of the environment that is being digitally visited.For example, simulating the hot sun of the Gobi Desert or the warm seabreeze of Hawaii's beaches. Dating—simulating a method of signaling apotential dating match, such as by simulating a blown kiss.Architecture, design and real estate—the ability to simulate the use ofan object that requires air flow to enhance the simulation. For example,designing or test driving a new motor cycle design and creating theunique experience of driving the motorcycle. Education—the haptic towersystem will help reinforce learning of various subjects, making learninga visceral experience, as opposed to relying on the traditional methodsof rote memorization. E-commerce—the ability to experience how a pieceof clothing looks and feels in a certain temperature or air flowenvironment. For example, a specific piece of clothing that looksparticularly good with a light breeze or movement by the user can beexperienced in the particular setting. This would allow the user toexperience the item in the particular setting without having to purchasethe item and physically wear or use it in the setting.

It is another object of the invention to provide for a system and methodthat triggers or controls at least one of a modulation (actuation orhaptic effect, for instance) for a peripheral device based on computervision processing of audio or video signals from an unscriptedprogramming feed. As a result, obviating content hurdles and opening thefull library of a/v based programming in communication with a peripheraldevice. In one aspect, the system may process at least one of an audioor video input for direct integration actuation or haptic effect from aperipheral device. The peripheral device may be in physical contact witha user or free from the user and in direct integration with an originalprogramming feed or live feed comprising native audio or video input.The system may further comprise a processor; a memory element coupled tothe processor; a program executable by the processor to: recognize atleast one of the native audio or video input from the originalprogramming feed or live feed, and determine for at least one taggedevent, at least one of a pixel color score, a pixel velocity score, anevent proximity score or an audio score; and convert the at least onescored event into at least one of an actuation output command or ahaptic output command and based on the output command, trigger orcontrol at least one of a haptic effect or actuation for the peripheraldevice in physical contact or free from the user and in directintegration with the original programming feed or live feed comprisingthe native audio or video input, whereby the user is not limited to alibrary of content wherein each content is coded with distinct actuationor haptic triggers corresponding to the content and direct integrationwith any audio or video content for at least one of an actuation orhaptic effect is enabled.

In one other aspect, a method is provided for processing at least one ofan audio or video input for direct integration actuation or hapticeffect from a peripheral device. The method may comprise the steps of:First, recognizing at least one of the native audio or video input froma feed, and determining for at least one tagged event, at least one of apixel color score, a pixel velocity score, an event proximity score oran audio score and finally; converting the at least one scored eventinto at least one of an actuation output command or a haptic outputcommand and based on the output command, triggering or controlling atleast one of a haptic effect or actuation for the peripheral device indirect integration with the feed comprising the native audio or videoinput. Content no longer needs to be limited to within provider anddeveloper silos in order to be coupled to a fully immersive experience.

Another aspect of the invention is a method for controlling a lighteffect based on a rendering of a web-page providing a script layerconfigured for scripting more customized effects from end peripheraldevices, enabling a more personalized and immersive experience.Generally, the method entails the steps of: Providing a web-browser pageinterface configured for script input for adjusting any one of an aspectof the immersive light effect from the at least one LEPD; rendering thescript-inputted web-browser page to an off-screen buffer visualized asat least a two-dimensional effects plane; applying a geo-positionaltransform and scaling of virtual LEPD's within the effects plane andcapturing at least a region of the rendered webpage; and controlling alight effect emitted from the at least one LEPD corresponding to theeffects plane transformed/scaled virtual LEPD and captured region of therendered web-browser page.

In yet other aspects, the rendered web-browser page for applying thetransform/scaling may be based on any of, or combination of, acustomizable script layer, standard UI layer, static or dynamic content.A system for rendering/transforming/scaling in the delivery of thecustomizable/immersive effects experience may also be provided, whereinthe system comprises a rendering module and a transform/scale module,and optionally, a peripheral device controller or control system.Collectively, the system or method provides for coordinated delivery ofeffects onto an end peripheral device based on a captured region andtransformed/scaled application of a rendered web-browser page. The endperipheral device, or effects-emitting peripheral device (EEPD), mayencompass the LEPD (light-diode strip, bulb), along with any one of ahaptic device, such as a controlled air-dispensing device, haptic vest,etc. As a result, an end-user, with just minimal low-level coding, mayscript customized and complex effects that deliver an immersive physicalexperience mirroring a grabbed region of the rendered, transformed, andscaled web-page.

In one other aspect, a solution is provided for countering issues offalse positives and latency of effects from an end-device during aviewing/gaming experience-both of which severely restrain a user fromexperiencing total immersion. To this end, the solution additionallyoffers options for delivery of complex combinatorial/layered effectsthat may be rippled from end-devices mirroring/corresponding to keyelements/events from the engaged content. The solution comprises asystem for end-device modulation by a hybrid trigger comprising: atleast one end-device (E-D) in communication with at least a first device(D1) outputting audio/video programming; a processor; a memory elementcoupled to the processor; a program executable by the processor to:position a virtual representation of the E-D on a digital canvasdisplayed on a D1-coupled display representing a user's physical andvirtual space; and capture from a corresponding region of the virtualspace for modulating an effect on a corresponding portion of the E-Dbased on a combination of at least two different triggers recognizingthe a/v element and/or a/v event.

In another aspect of the hybrid trigger approach for end-devicemodulation, a method is provided comprising the steps of: positioning avirtual representation of the E-D on a digital canvas displayed on aD1-coupled display representing a user's physical and virtual space; andcapturing from a corresponding region of the virtual space formodulating an effect on a corresponding portion of the E-D based on atleast one of a triggering a/v element or triggering a/v event trigger.

Other aspects and advantages of the invention will be apparent from thefollowing description and the appended claims.

BRIEF DESCRIPTION OF DRAWINGS

The drawings illustrate the design and utility of embodiments of thepresent invention, in which similar elements are referred to by commonreference numerals. In order to better appreciate the advantages andobjects of the embodiments of the present invention, reference should bemade to the accompanying drawings that illustrate these embodiments.

However, the drawings depict only some embodiments of the invention, andshould not be taken as limiting its scope. With this caveat, embodimentsof the invention will be described and explained with additionalspecificity and detail through the use of the accompanying drawings inwhich:

FIG. 1 is a front perspective view diagram of an apparatus in accordancewith an aspect of the invention.

FIG. 2 is a block diagram of the air flow configuration in accordancewith an aspect of the invention.

FIG. 3a is a block diagram of the cooling temperature feedback loop inaccordance with an aspect of the invention.

FIG. 3b is a block diagram of the heating temperature feedback loop inaccordance with an aspect of the invention.

FIG. 4 is a system diagram of the system configuration in accordancewith an aspect of the invention.

FIG. 5 is a method flow diagram of the method of delivering haptics inaccordance with an aspect of the invention.

FIG. 6 is a system block diagram of the haptic engine in an exemplaryenvironment according to an aspect of the invention.

FIG. 7 is a system block diagram of the haptic engine isolated inaccordance with an aspect of the invention.

FIG. 8 is a system block diagram in accordance with an aspect of theinvention.

FIG. 9 is an interaction flow diagram in accordance with an aspect ofthe invention.

FIG. 10 is an interaction flow diagram in accordance with an aspect ofthe invention.

FIG. 11 is a process flow diagram according to an aspect of theinvention.

FIG. 12 is a method flow diagram in accordance with an aspect of theinvention.

FIG. 13 is a system block diagram of the peripheral modulation of theunscripted feed in accordance with an aspect of the invention.

FIG. 14 is an interaction flow diagram of the peripheral modulation ofthe unscripted feed in accordance with an aspect of the invention.

FIG. 15 is a method flow diagram of the peripheral modulation of theunscripted feed in accordance with an aspect of the invention.

FIG. 16 is a method flow diagram of the light-emitting peripheralmodulation of the unscripted feed in accordance with an aspect of theinvention.

FIG. 17 is an exemplary process flow diagram illustrating thecustomizable and immersive effects delivery in accordance with an aspectof the invention.

FIG. 18 is a screenshot illustrating an exemplary script layer on aweb-browser page for delivery of the customizable and immersive effectsin accordance with an aspect of the invention.

FIG. 19 illustrates a block diagram of an exemplary system in accordancewith an aspect of the invention.

FIG. 20 is an exemplary process flow diagram illustrating thecustomizable and immersive effects delivery in accordance with an aspectof the invention.

FIG. 21 illustrates a block diagram of an exemplary system in accordancewith an aspect of the invention.

FIG. 22 illustrates a block diagram of an exemplary system in accordancewith an aspect of the invention.

FIG. 23 is a method flow diagram in accordance with an aspect of theinvention.

FIG. 24 is a method flow diagram in accordance with an aspect of theinvention.

FIG. 25 is a screen-shot illustrating an exemplary layer of thehybrid-triggered effects in accordance with an aspect of the invention.

FIG. 26 is a screen-shot illustrating an exemplary layer of thehybrid-triggered effects in accordance with an aspect of the invention.

FIG. 27 is a screenshot illustrating an exemplary layer of thehybrid-triggered effects in accordance with an aspect of the invention.

FIG. 28 is a screenshot illustrating an exemplary layer of thehybrid-triggered effects in accordance with an aspect of the invention.

FIG. 29 is a screenshot illustrating an exemplary layer of thehybrid-triggered effects in accordance with an aspect of the invention.

FIG. 30 is a screen-shot illustrating an exemplary layer of thehybrid-triggered effects in accordance with an aspect of the invention.

FIG. 31 is a screen-shot illustrating an exemplary layer of thehybrid-triggered effects in accordance with an aspect of the invention.

DETAILED DESCRIPTION

In the following description, for purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the invention. It will be apparent, however, to oneskilled in the art that the invention can be practiced without thesespecific details.

Reference in this specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least one embodimentof the invention. The appearances of the phrase “in one embodiment” invarious places in the specification are not necessarily all referring tothe same embodiment, nor are separate or alternative embodimentsmutually exclusive of other embodiments. Moreover, various features aredescribed which may be exhibited by some embodiments and not by others.Similarly, various requirements are described which may be requirementsfor some embodiments but not other embodiments.

The present embodiments disclose apparatus, systems and methods forallowing users to receive targeted delivery of haptic effects—air flowof variable intensity and temperature—from a single tower or surroundtower configuration. Each tower housing may have an integrated fanassembly creating air flow of variable intensity, along with anintegrated temperature element within a duct, which treats the air flowwith variable temperature. The haptic tower may have an enclosed,modular assembly that manipulates air flow, fluid flow, scent, or anyother haptic or sensation, for an immersed user. The fan assembly may beany device for producing a current of air by movement of a broad surfaceor a number of such surfaces. The duct may be any channel, tube, pipe orconduit, by which air, fluid, scented air, or any other substances maybe conducted or conveyed—and may or may not house the temperatureelement. The temperature element may be a heat exchanger that changesthe temperature of air, fluid, scented air, or any other substance.Moreover, the system has an application of sensor technology to capturedata regarding a user's body positioning and orientation in the realenvironment. This data, along with the data from a program coupled tothe system, is relayed to the micro controller with instructions codedthereon to instruct the relevant towers to direct air flow, variableintensity of air flow, variable temperature of air flow, and targeteddispensing of haptic effect. These features expand the sense of realismand immersion of a user in a virtual space. Various other back-endfunctionalities may be taken advantage of by a user through aninteractive mobile app or from the high-resolution, easy-to-useuser-interface display. Aside from the sophisticated components andelectronics delivering precision haptics, the intelligent andcontextually-aware system also easily integrates with any home automatedsystem via Wi-Fi, ZigBee, or Bluetooth 4.0. The system also easilyconnects to a cloud-based server allowing it to interface with themobile app, enabling the user to choose from a variety of informativedashboard alerts and features. Moreover, a peer-sharing tool allows forusers to share aspects of their immersive experience.

With reference now to the drawings, and in particular to FIGS. 1 through5 thereof, a haptic delivery apparatus, system, and method embodying theprinciples and concepts of the present invention and generallydesignated by the reference numeral 100 will be described.

FIG. 1 is a front perspective view diagram illustrating an apparatus forthe automated dispensing of targeted and precise haptics, in accordancewith one embodiment of the present invention. A housing unit 100dispenses air of precise air flow and temperature to targeted portionsof a user based on data from a user in the virtual and real space. Inthe present example, the housing unit 100 may be a haptic tower 408resting on the floor or a countertop device, configured to house a fanassembly 102, but any number of fan assemblies 102 may be added, withoutdeparting from the scope of the invention. Likewise, while in thepresent example, the housing unit 100 may have a separate shutter 106,duct 108, and temperature element 110, depending on the desiredtemperature range, any number of shutters 106, ducts 108, andtemperature elements 110 may be used, without departing from the scopeof the invention. Other embodiments may be a stand-up haptic tower 408,although any size housing unit 100 is disclosed, including smaller,portable devices for on-the-go individual use, or larger units, withincreased number of system components or more industrial strengthcomponents, appropriate for group applications.

The preferred embodiment of the housing unit 100 may have an integratedfan assembly 102, motor output 104, shutter 106, duct 108, temperatureelement 110, dispensing nozzle 112, rotatable base 114, and interfacedisplay 116. Housing unit 100 may encompass a housing top wall 118,bottom wall, and side walls 122, 124 that wrap around to meet the frontwall 126 and back wall 128. Front wall 126 may have a dispensing nozzle112 for targeted delivery of precise haptics onto a user. Front wall 126may also have a user interface display 116 for mediating userinteraction with dispensing device.

In other embodiments, though not shown in FIG. 1, the housing unit 100may have flat side walls 122, 124 and flat front and back walls 126,128. The front wall 126 may have a dispensing nozzle 112 hidden behind aflush wall with the means of opening and closing. The dispensing nozzle112 may have separate outlets for air, fog, and mist. Additionally, thedispensing nozzle 112 may have the ability to rotate, or change thediameter of the inlet, in order to target the direction of the air flow,as well as alter the intensity of the air flow. Although not shown inFIG. 1, the housing unit 100 may have a front wall 126 void ofdispensing nozzles, rather, the haptic delivery may be via a ventsystem, or any other outlet. The front wall 126 may be void of the userinterface display 116, and rather, may be included in the mobile deviceapplication.

In further detail, still referring to FIG. 1, a housing unit 100 mayhave a rotatable base 114, which may pivot the housing unit 100 in atleast one axis of motion. A rotating base 114 allows for the housingunit 100 to rotate on its base to allow for more targeted delivery ofhaptic effects. More particularly, a rotating base 114 may allow for thehousing unit 100 to rotate on its base in at least one axis of motion toprovide for a panning air flow effect. In other embodiments, therotatable base 114 may allow for motion along multiple axis of rotation.In one embodiment, pivoting and targeted haptic delivery may be furtherenhanced by using head tracking or full body tracking system. Otherembodiments may include a housing unit 100 with a dispensing nozzle 112,the pivoting and rotation of which may be also enhanced with theaddition of head tracking or full body tracking systems.

With continuing reference to FIG. 1, a housing unit 100 may include auser interface display 116, wherein the user interface may be integratedas a built-in console display. While in the present example, a built-inconsole display is shown, any type of user interface display 116 may bedisclosed, including a mobile device display, a wearable device display,monitors, or any type of access device, without departing from the scopeof the invention. In a preferred embodiment, the user interface display116 may include a display page for receiving a request for a hapticoutput selection. The request being from a menu, a haptic suggestionengine, or user-initiated. The display page may then prompt a user toconfirm the request. Other embodiments may include a display page thatdoes not require a user to confirm the request, and instead, signalsconfirmation of the request and initialization.

Alternate embodiments may involve a user interface display 116authenticating a user by any form of short-range or long-range wirelessprotocol standards, without departing from the scope of the invention.In authenticating a user, an authentication module may be further causedto recognize the user device at a particular haptic tower housing aunique short-range communication tag. The module may identify andauthenticate the particular tower and user device by recognizing theunique tag, and then, authenticate the user by identifying the userdevice located at the particular tower. The unique, short-range tag maybe a NFC tag, RFID chip, Bluetooth, ZigBee, or any short-range orlong-range communication protocol standard. Additional methods ofauthentication may be accomplished via user input.

In yet another embodiment, the user interface display 116 may include avoice—activated request option receiving a request voice command,whereby the request voice command may be in communication with avoice-activated module querying at least one pre-defined database basedon the request voice command. The voice-activated module may be incommunication with a natural language module, whereby the request voicecommand is sent from the voice-activated module to the natural languagemodule. The natural language module may be configured to convert therequest voice command into a haptic output instruction querying at leastone pre-defined database based on the request voice command.

In yet another embodiment, the user interface display 116 may receive arequest voice command for a haptic output selection and interact with auser via voice response by having a voice activated module incommunication with the natural language module and the voice activatedmodule in communication with a voice response module, whereby the voiceresponse module alerts the user of the various stages of the hapticoutput selection via the voice-activated user interface using naturallanguage to describe the various stages of processing, from anintroduction and identification of a user; to a haptic output selectioninquiry or request or suggestion; to confirmation of a haptic outputselection; and finally, initialization.

Still referring to the user interface display 116, the user maycalibrate the maximum and minimum temperatures based on the user'spreference. For example, if the user is hot, the user may calibrate thesystem to emit only cool air and not activate the hot side at all, andvice versa, for a cold user. If the user does not want to have airhaptic sensation as a part of the virtual experience, the user mayoverride the software experience and use the system as a normal heating,cooling or fan system. A manual system override feature may be presenton the interface display 116 for the haptic system control.

Although not shown in FIG. 1, some embodiments may include a housingunit 100 that includes an air bursting effect system. The air burstingeffect system delivers high velocity air flow directed at the user.According to one embodiment, the air bursting effect is created by theuse of air vortices. Rather than using a manually actuated bag attachedto bungee cord, a handle may be attached to an actuating rod supportedby a rail system powered by a motor assembly. The rail system may have aspur gear with only half the teeth around the perimeter so that when therack on the slider is no longer in contact with the gear teeth, theslider is pulled forward by the spring.

In other embodiments, an array of miniature speakers to create a largeenough volume of air displacement within a chamber to generate aminiature air vortex may be used. Another air bursting effect system maycreate air displacement via the use of a larger speaker or a sub-woofer.Some embodiments may include creating air bursting effects through theuse of compressed air. Using an air compressor with an air tank, fittedwith an electro mechanical valve, aimed at the user, a burst ofcompressed air can be used to enhance the users sense of presence. Avariable controlled electro mechanical valve can vary intensity of airflow and pressure. While in the present examples, the air burstingeffect system may be integrated within the housing unit 100, airbursting effect systems not integrated within the housing unit 100, butrather, as a separate unit is disclosed, without departing from thescope of the invention.

Although not shown in FIG. 1, in yet another aspect of the invention, ahousing unit 100 may include a fog and mist dispensing system. In anexemplary embodiment, a sprinkler or misting system may be connected toa water pump attached to an electric controlled check valve to allow theprecise release of water in a mist-like fashion. In another embodiment,the fog and dispensing system may include at least one fluid supply linein fluid communication with at least one fluid supply and with at leastone outlet; condensing means for air and fluid from the fluid supply;and dispensing fog or mist via an outlet. In some embodiments, theoutlet may be a dispensing nozzle 112 or vent. In some other aspects,the fluid supply line may be in direct communication with the dispensingnozzle 112 emitting air flow as well, or the fluid supply line may be indirect communication with a dispensing nozzle 112 exclusive to the fogor mist. In other embodiments, water misting device or water jetattachment subsystems can be attached to the haptic tower, much like themodularized air bursting effect systems attached to the haptic tower.Using misting systems connected to a water pump attached to an electriccontrolled check valve, the system may allow for the precise release ofwater in a mist like fashion. Likewise, in some embodiments, a scentsystem including a scented-air supply connected to a pump, attached toan electric controlled check valve, may allow for the precise deliveryof scented-air.

FIG. 2 shows a block diagram of the air flow configuration in accordancewith one embodiment of the invention. The fan assembly 202, controlledby a motor output 204, creates air flow of variable intensity, and theair flow is directed through either hot, cold, or ambient shutters 206,whereby the air is directed through a respective temperature duct 208.Air flow is treated with variable temperature by a temperature element210. Air flow of variable intensity and temperature are then directedout of an outlet 212.

In one exemplary embodiment, the fan assembly 202 may be a blower fan(also known as a squirrel cage) to produce a smooth, uniform flow ofair. Traditional axial desk fans “chop” the air up and produce anon-uniform flow of air, which is not ideal for this application. Themotor output 204 powering the blower fan assembly 202 will have avariable controlled speed output. In other exemplary embodiments, thefan assembly 202 will be an impeller design, or any design that maycreate a smooth, uniform flow of air. Other embodiments may include abrake for tight control of the output air flow from the fan assembly202. Airflow will have a range of approximately 0 to 200 CFM.

In yet another exemplary embodiment, the air flow is directed tospecific shutters 206, whereby it is channeled into respective ducts208, and appropriately treated with temperature by temperature element210. Servo motors may control dampers or flat shutters 206, and theseshutters 206 will open and close, controlling the air flow throughdifferent temperature ducts 208. After redirecting the air into one ofthe three separate ducts 208, each duct 208 has either a hot, cold or notemperature element 210. After redirecting the air into one of the threeseparate ducts 208, each duct 208 has either cold, hot or no temperaturetreatment to the out-flow of air. For heated air, the air flows throughthe “hot” duct 208 with an exposed heating element 210. In a preferredembodiment, the air may flow through an exposed Positive TemperatureCoefficient (PTC) ceramic heater element, or any thermistor with a highnon-linear thermal response, such as barium titanate or lead titanatecomposites. In other embodiments, the heating element 210 may be acondenser heat sink in a vapor-compression cycle, thermoelectric heatingusing Peltier plates, Ranque-Hilsch vortex tube, gas-fire burner, quartzheat lamps, or quartz tungsten heating, without departing from the scopeof the invention. For the “cold” duct 208, the air flows through acooling element 210. In a preferred embodiment, the air may flow througha traditional finned air conditioning evaporator in a vapor-compressioncycle. Alternate embodiments of the cooling element 210 may includethermoelectric cooling using the Peltier effect, chilled water cooler,Ranque-Hilsch vortex tube, evaporative cooling, magnetic refrigeration,without departing from the scope of the invention. For the ambient duct208, air bypasses both the heating and cooling temperature elements 210.In alternate embodiments, the air from the fan assembly 202 is directedinto a single temperature duct 208, where the air is exposed to bothheating and cooling temperature elements 210 integrated into the singletemperature duct 208. Other embodiments may include heating or coolingthe air flow into any number of shutters 206, temperature ducts 208, andtemperature elements 210, without departing from the scope of theinvention.

FIG. 3a shows a block diagram of the temperature feedback loop for acooling element. In one preferred embodiment, the finned condenser orany cooling element 304 requires a temperature sensor 302, such as athermocouple, in contact with each cooling element 304 to monitor thetemperature of the temperature ducts 306. These temperature sensors 302may be an infrared sensor, bimetallic thermocouple sensor, pressurespring thermometers, or infrared camera. Any one of these temperaturesensors 302 may keep the temperature of the temperature ducts 304 at aconstant temperature, through a feedback loop with the micro controlboard. In this exemplary embodiment, a thermostat 308 is set to aspecific temperature range that it desires to reach using softwaresignals from the CPU 406. The thermostat 308 then measures thetemperature using the temperature sensor 302. Based on what the measuredtemperature is compared to the set temperature range, the thermostat 308acts as a relay device that sends an on/off signal to the coolingcompressor 310 to turn on/off. As more air flows through the coolingelement 304, more air is cooled and the cooling element 304 will heat upin temperature, triggering the thermostat 308 to turn on the coolingcompressor 310.

FIG. 3b illustrates a block diagram for the temperature feedback loopfor a heating element. In accordance with an exemplary embodiment, thethermostat 312 is set to a specific temperature range that it desires toreach using software signals from a CPU 406. The thermostat 312 thenmeasures the temperature using the temperature sensor 314. Temperaturesensors 314 may include infrared sensors, bimetallic thermocouplesensors, pressure spring thermometers, or infrared cameras. Based onwhat the measured temperature is compared to the set temperature range,the thermostat 312 acts as a relay device that sends an on/off signal toa switch that allows current to flow through the heater element 316which heats the heater element 316. As more air flows through the heaterelement 316, more air is heated and the heating element 316 will cooldown in temperature, triggering the thermostat to power the heaterelement.

Pre-heated and pre-cooled temperature ducts 304, in combination withshutters 206, will help maintain the low latency of the virtualenvironment demands. Low latency of the environmental simulation isimportant to the experience of the user because when the user seesvisual cues, the environmental simulator needs to respond immediately,otherwise a lag between the sense of feeling and environment can have anundesirable effect. Latency is the interval between the stimulation andresponse, or also known as the time delay between the cause and effectof some physical change in the system being observed. For example, theuser raises his arm in the physical world and his arm in the virtualworld raises with an obvious delay representing high latency of thesystem.

FIG. 4 shows a system diagram of the surround haptic systemconfiguration. In one general aspect of the invention, a system maycomprise of a sensor or a series of sensors 402 to detect a user's bodyposition and orientation. Although not shown in FIG. 4, in otherembodiments, a higher resolution of data capture related to userposition and orientation may be achieved using body-tracking,hand-tracking, head-tracking, or eye-tracking sensors. Tracking enablesthe measuring of simple behaviors of a user in the physical world. Forexample, the user took one step forward in the physical world and thedistance of one step was measured and tracked by a computer system withprecise coordinates. When tracking, data is available in the systemcomputer, it can be used to generate the appropriate computer-generatedimagery (CGI) for the angle-of-look at the particular time. For example,when a user's head is tracked, the computer system renders thecorresponding computer-generated imagery to represent the digital world.Examples of tracking may be the use of a depth sensing camera for handtracking; electromagnetic motion tracking for limb and body tracking;LED array tracking; accelerometer tracking; eye tracking; andeye-tracking with infrared and near-infrared non-collimated light tocreate corneal reflections. Audio sensor data may also be a part of theuser input data.

Another feature to enhance presence is to control the direction of thehaptic tower 408 using motors which allow haptic towers 408 to pivot inplace by its rotatable base 114 and most mimic the virtual environmentthe user is in. This can be further enhanced by using head track or fullbody tracking. This body tracking may also be used for the control andaiming of the rotatable dispensing nozzle 112 at particular track bodylocations. Additionally, in an alternative embodiment, spacializationsoftware within the virtual experience with adaptive algorithms maychange the intensity of air flow based on tracking of the users positionin the virtual space. These features effectuate targeted delivery ofhaptic effects, enhancing the immersive VR experience for the user.

In other embodiments, user environment sensors, either attached to theuser or placed near the user, give the system an initial temperaturereading to customize the experience to the user's environment state. Forexample, if the ambient temperature near the haptic towers 408 is cold,the system can compensate by setting the temperature experience to omitcold temperature output. In yet another embodiment, flow sensors at theuser's location or at the outlet of the haptic towers 408 measure andcontrol the flow output of the fan assembly 202, mist output and burstoutput. Alternative embodiments may include measuring the flow output ofthe fan assembly 202 by measuring the rotating speed of a motor in a fanassembly 202. Other embodiments include audio sensor data as being apart of the user input data.

Still referring to FIG. 4, the user data captured by the sensor orsensors 402 related to user body position and orientation, may becommunicated to the micro controller 404, which will relay input signalsfrom sensors 402 and relay output commands to the haptic towers 408, viaa CPU 406. The micro controller 404 may be a small computer or a singleintegrated circuit containing a processor core, memory and programmableinput. The micro controller 404 codes the data from the CPU 406,including user data from the sensors and program content data, toactuate the haptic towers 408 to deliver the haptic effects. In oneembodiment, system configuration may include haptic towers 408 thatwirelessly communicate with the CPU 406 through any short-range mode ofwireless communication, such as Wi-Fi, UWB, Bluetooth, ZigBee, or anyprotocol standards for short range wireless communications with lowpower consumption. Each haptic tower 408 may send and receive commandsto the CPU controlling the experience. Another embodiment of the systemmay have the haptic towers 408 connect to the CPU 406, directly withouta micro controller 404, through USB, or any cable, connector andcommunication protocols used in a bus for connections, communications,and power supply for electronic devices. The CPU 406 would communicatedirectly with each haptic tower 408 sending and receiving data incoordination with the sensor user data and coded experience data. Thisconfiguration would have each haptic tower 408 powered independently orthrough a power controller where each additional haptic tower 408 wouldconnect to the power controller.

In another configuration, the flow of data communication may be thethrough a wired connection where each haptic tower 408 would be wired toa micro controller 404, and the micro controller 404 is wired to the CPU406, through USB, or any cable, connector and communication protocolsused in a bus for connections, communications, and power supply forelectronic devices. The haptic towers 408 would send sensor data to themicro controller 404, which would relay the data to the CPU 406. The CPU406 would interpret the data and respond accordingly by sending commandsto the micro controller 404, which would relay the commands to theassociated haptic tower 408. In yet another embodiment, the haptictowers 408 may wirelessly communicate with the micro controller 404,bypassing the CPU 406, by any of the known method of short-rangewireless connection, such as Wi-Fi, UWB, Bluetooth, ZigBee, or anyprotocol standards for short range wireless communications with lowpower consumption. Each haptic tower 408 can be powered through themicro controller 404, or independently powered. The micro controller 404may be placed on a computer desk near the CPU 406. A USB connection mayconnect the micro controller 404 to the CPU 406. Additionally, a powercord may be plugged into a standard AC120V socket, which is attached tothe microcontroller 404. In one embodiment, the haptic tower 408 mayhave a power cord or control wire that will plug into the microcontroller 404. While in the present example, the haptic tower 408 andmicro controller 404 are networked via a cord or wire, other embodimentsmay include communicating over wireless short-range or long-rangenetworks.

In one preferred embodiment, not shown in FIG. 4, a high-levelinitialization protocol may begin with establishing a micro controllerand CPU connection and confirming power of the micro controller. Inanother embodiment, the system may establish connection with each haptictowers' individually in a configuration void of a micro controller hub.Next, the initialization protocol may confirm if each haptic tower isupright and in the right orientation; read initial temperature readingsfrom all thermometers; confirm user positioning—location relative tohaptic towers; read initial positions of all servo motors, dampershutter motors, tower positioning motors, nozzle motors; confirm motorsare operational; next, set all servo motors to default positions;confirm motor positions with output position; confirm minimum distancebetween user/object and the haptic tower outlet; confirm thefunctionality of the heating and cooling temperature elements; confirmwith thermometer reading max/min temperature with the max power toheating/cooling element relative to room temperature; then, safelyconfirm no overloading of circuitry or overheating; and confirm fanmotor functionality and confirm command speed with tachometer inputspeed.

In another preferred aspect, also not shown in FIG. 4, a high-levelcommunication protocol may include a CPU communicating with a haptictower library to create a programmed experience of specific outputhaptics. The CPU may then send instructions to a haptic tower microcontroller (MCU) via USB, USCI, I2C, SPI, UART, or other wirelesscommunications protocols, which may, in turn, coordinate actuation ofmotors in series, or in parallel, to deliver the latent-free hapticexperience. The use of a micro controller hub, as opposed to a haptictower micro controller, may also be used to coordinate function ofmotors, without departing from the scope of the invention. The hapticmicro controllers may drive actuation of motors using pulse-widthmodulation (PWM). Pulse-width modulation signals result in latent-freeresponses and allow for variable control of a driver and actuator.

More particularly, still referring to a preferred embodiment of thecommunication protocol, simultaneous control of the haptic experiencewill be integrated into the onboard micro controller (MCU). For example,the CPU sends the coordinates of the haptic experience to the MCUthrough a dedicated communication line. The combination of predictivealgorithms integrated into the MCU and the communication protocol fromthe CPU, allows the MCU to predictively lower haptic experience latencyto generate a unique and specific entertainment experience. The MCU isconfigured to interpret the positional data and simultaneouslycoordinate the actuator array to precisely deliver the haptic output.Typical CPU loads are high due to the graphical intensity and computingpower required to create low latency virtual reality experience. As aresult, allowing the MCU to interpret and drive the haptic experience inan autonomous manner offloads the CPU requirements and decrease latencybetween the visual image and haptic experiences. Alternatively, seriescontrol of the haptic experience may be integrated into the on-board MCUto off-load CPU demands and decrease latency as well. An additionaldedicated communication line between the CPU and on-board MCU may embodythe user profile and contextual information. This user profile andcontextual information may be a combination of data points ranging fromlocal weather, wearable temperature data, user preferences, user healthdata, etc. This data may then be used to augment the sensor data andcontent data to drive an even more personalized haptic experience—in alow-demand and low latency environment.

While not shown in FIG. 4, in yet another configuration of thecommunication protocol, the on-board MCU may be an autonomous powermanagement tool that can ultimately determine the power requirements foreach element. For example, if specific haptic towers will not requirethe cooling requirement, the MCU can autonomously control the powersupply to the cooling temperature element. This improves the overallpower efficiency of the system without losing the required low latencyexperience. Another embodiment of a communication protocol may be for acomprehensive safety monitoring system. Each haptic tower is fitted withmoving motors, heating and cooling temperature elements that can createa number of hazards. The continuous communication between the CPU andMCU is required due to a need to protect the user from any hazard.Continuous monitoring of circuit behavior, thermometers, motor output,and complex simultaneous and series systems are important for usersafety and hazard mitigation. This dedicated line will communicate witha dedicated line to ensure the CPU knows when to halt any virtualexperience and draw attention to the user in case of an emergency in theform of a dashboard alert formatted for an interface display.

According to one embodiment, the system will be a modular surroundhaptic system, as shown in FIG. 4. The system may include either the twoor four haptic tower 408 configurations with a micro controller 404controlling all of the haptic towers 408. The user may then set up eachhaptic tower 408 approximately three feet distance from the user's torsodepending on how many haptic towers 408 are set up. The user may orienteach haptic tower 408 such that the air outlet or dispensing nozzle 112may be pointed towards the user's torso/head area. In some embodiments,height may be adjustable via either sliding the system up or down on atripod system. The user may be able to manually adjust the dispensingnozzle 112 direction in the desired angle for the user. Automatedhead/body tracking may allow the system to automatically aim at theuser. Some embodiments may include haptic towers that move dynamicallywithin a confined space to simulate wind or other air displacement frommultiple points of origin, greatly expanding the degree of locationalspecificity, as compared to static towers. Alternate embodiments mayinclude system configurations with any number of haptic towers,featuring at least a single haptic tower.

In some aspects of the invention, the location of the individual haptictowers 408 within the surround system configuration may be calibrated.Software and hardware may recognize the location of each haptic tower408 to accurately simulate the virtual environment. The location may befixed for each haptic tower 408, where each haptic tower 408 will bemanually labeled with a location of where that haptic tower 408 isintended to be oriented relative to the user. In another aspect,calibration of the location of each haptic tower 408 may not need afixed set location, rather the user may set each haptic tower 408 to alocation using software confirming each haptic tower location. In yetanother aspect, calibration of tower location may be automated,obviating the need for user input. In continuing reference to FIG. 4, asystem may include an interactive display, wherein the interactivedisplay may be any one of the following: a head-mounted display; adisplay screen; a 3-D projection; and a holographic display.

While not shown in FIG. 4, embodiments may include the addition of aremote server to provide for back-end functionality and support. Theserver may be situated adjacent or remotely from the system andconnected to each system via a communication network. In one embodiment,the server may be used to support verification or authentication of auser and a mobile device application function. In authenticating a user,a server may be further caused to recognize the user device at aparticular system component, whether it is a haptic tower, microcontroller, or any other system component that may be able to house aunique short-range communication tag. The server may identify andauthenticate the particular component and user device by recognizing theunique tag, and then, authenticate the user by identifying the userdevice located at the particular component. The unique, short-range tagmay be a NFC tag, RFID chip, Bluetooth, ZigBee, or any short-rangecommunication protocol standard. The remote server may be furtherconfigured to support a user haptic output history function; helpsupport a network sharing function; and support a haptic outputselection search engine. The remote server may be further configured toprovide a user-control system, which authenticates the user andretrieves usage data of the user and applies the data against predefinedcriteria of use.

Other embodiments may include a remote server that is configured toprovide a contextually-aware haptic output suggestion engine, which mayaccess the user haptic output history function and at least one usercontextual information to cause the processor to display a suggestedhaptic output on at least one display interface 116. Provisioning of theremote server may be delivered as a cloud service. In yet otherembodiments, a haptic tower 408 may be associated with an Internet ofThings, whereby the haptic tower 408 is fully integrated into a user'shome automation system, thereby providing additional contextualinformation for a contextually-aware haptic output suggestion engine.

FIG. 5 shows a method flow diagram for the method of delivering preciseand targeted haptic effects of variable air flow and temperature to auser. The preferred components, or steps, of the inventive method are asfollows: first, in step 1502, sensor or sensors 402 may detect userposition and orientation. The user data captured by the sensor orsensors 402 related to a user body position and orientation, may becommunicated to the micro controller 404, which relays the signal to theCPU 406. Alternatively, a higher resolution of data capture related touser position and orientation may be achieved using body-tracking,hand-tracking, head-tracking, or eye-tracking sensors. Tracking enablesthe measuring of simple behaviors of a user in the physical world, inorder to virtualize the user and further actuate rotation of the base,as well as nozzles 112, for precise and targeted delivery of hapticsonto a user. Examples of tracking may be the use of a depth sensingcamera for hand tracking; electromagnetic motion tracking for limb andbody tracking; LED array tracking; accelerometer tracking; eye tracking;and eye-tracking with infrared and near-infrared non-collimated light tocreate corneal reflections. Audio sensor data may also be a part of theuser input data.

Step 2 504, user data may be communicated to the micro controller 404and then communicated to the haptic towers 408. The micro controller 404may code the data from the CPU 406, including user data from the sensors402 and program content data, to actuate the haptic towers 408 todeliver the haptic effects. The user data captured by the sensor orsensors 402 related to user body position and orientation, may becommunicated to the micro controller 404, which relays the signal to theCPU 406. The micro controller 404 codes the data from the CPU 406,including user data from the sensors 402 and program content data, toactuate the haptic towers 408 to deliver the haptic effects. Oneembodiment may include haptic towers 408 that wirelessly communicatewith the CPU 406 through any short-range mode of wireless communication,such as Wi-Fi, UWB, Bluetooth, ZigBee, or any protocol standards forshort range wireless communications with low power consumption. Eachhaptic tower 408 may send and receive commands to the CPU 406controlling the experience.

Another embodiment may have the haptic towers 408 connect to the CPU406, directly without a micro controller 404, through USB, or any cable,connector and communication protocols used in a bus for connections,communications, and power supply for electronic devices. The CPU 406would communicate directly with each haptic tower 408 sending andreceiving data in coordination with the sensor user data and codedexperience data. This configuration would have each haptic tower 408powered independently or through a power controller where eachadditional haptic tower 408 would connect to the power controller.

In another configuration, the flow of data communication may be thethrough a wired connection where each haptic tower 408 would be wired toa micro controller 404, and the micro controller 404 is wired to the CPU406, through USB, or any cable, connector and communication protocolsused in a bus for connections, communications, and power supply forelectronic devices. The haptic towers 408 would send sensor data to themicro controller 404 which would relay the data to the CPU 406. The CPU406 would interpret the data and respond accordingly by sending commandsto the micro controller 404, which would relay the commands to theassociated haptic tower 408.

In yet another embodiment, the haptic towers 408 may wirelesslycommunicate with the micro controller 404, bypassing the CPU 406, by anyof the known method of short-range wireless connection, such as Wi-Fi,UWB, Bluetooth, ZigBee, or any protocol standards for short rangewireless communications with low power consumption. Each haptic tower408 can be powered through the micro controller 404, or independentlypowered. Alternatively, step 2 504 may involve a micro controller 404that only codes data from a program content data store in the CPU 406,and not require sensor 402 captured user data. The coded signal from themicro controller 404 actuates the haptic tower 408 to perform theprocess of delivering targeted air flow of variable intensity andtemperature. Still referring to FIG. 5, step 3 506 describes the microcontroller 404 instructing the haptic tower 408 to actuate a poweroutput to control variability of air flow rate. In a preferredembodiment, the micro controller 404 instructs the haptic tower 408 toactuate a motor 204 with variable controlled speed output for powering afan assembly 202. In alternative embodiments, the air flow results inair flow of variable intensity by the micro controller 404 instructingthe haptic tower 408 to actuate a valve in creating variable air flowrate. In yet another embodiment, a brake for tight control of the outputair flow from the fan assembly 202 may result in the variability of airflow rate. In yet another configuration of the communication protocol,the CPU may send the coordinates of the haptic experience to an on-boardmicro-controller (MCU) through a dedicated communication line. Thecombination of predictive algorithms integrated into the MCU and thecommunication protocol from the CPU, allows the MCU to predictivelylower haptic experience latency to generate a unique and specificentertainment experience. The MCU is configured to interpret thepositional data and simultaneously coordinate the actuator array toprecisely deliver the haptic output. As a result, allowing the MCU tointerpret and drive the haptic experience in an autonomous manneroffloads the CPU requirements and decreases latency between the visualimage and haptic experiences. Alternatively, series control of thehaptic experience may also be integrated into the MCU to off-load CPUdemands and decrease latency as well.

Step 4 508 describes a preferred embodiment of the method in which theair flow of variable flow rate may be directed into a specifictemperature duct 208 with the use of motored shutters 206. The air flowmay be directed to specific shutters 206, whereby it is channeled intorespective ducts 208, and appropriately treated by a temperature element210. Servo motors may control dampers or flat shutters 206, and theseshutters 206 will open and close controlling the air flow throughdifferent temperature ducts 208.

In continuing reference to FIG. 5, step 5 510 describes an exemplaryembodiment of the method in which air flow is directed into either atemperature duct 208 or ambient duct 208, depending on the need fortemperature treatment based on a data signal. If temperature treatmentis required, step 6 512 describes treating the air by a temperatureelement 210 in a respective duct 208. After redirecting the air into oneof the separate temperature ducts 208, each duct 208 has either a hot orcold temperature element 210. For heated air, the air flows through the“hot” duct 208 with an exposed heating element 210. In a preferredembodiment, the air may flow through an exposed Positive TemperatureCoefficient (PTC) ceramic heater element, or any thermistor with a highnon-linear thermal response, such as barium titanate or lead titanatecomposites. In other embodiments, the heating element 210 may be acondenser heat sink in a vapor-compression cycle, thermoelectric heatingusing Peltier plates, Ranque-Hilsch vortex tube, gas-fire burner, quartzheat lamps, or quartz tungsten heating, without departing from the scopeof the invention. For the “cold” duct 208, the air flows through acooling element 210. In a preferred embodiment, the air may flow througha traditional finned air conditioning condenser in a vapor-compressioncycle. Alternate embodiments of the cooling element 210 may include anevaporator heat sink in a vapor-compression cycle or thermoelectriccooling using the Peltier effect, chilled water cooler, Ranque-Hilschvortex tube, evaporative cooling, magnetic refrigeration, withoutdeparting from the scope of the invention. In alternate embodiments, theair from the fan assembly 202 is directed into a single temperature duct208, where the air is exposed to both heating and cooling temperatureelements 210 integrated into the single temperature duct 208. Otherembodiments may include heating or cooling the air flow into any numberof shutters 206, temperature ducts 208, and temperature elements 210.

Step 7 514 describes directing ambient air through a duct 208 without atemperature element 210. In alternate embodiments, the redirected airflow may be all directed into a single duct 208, regardless of therequirement for ambient or temperature treatment. In accordance, withthis embodiment, the air from the fan assembly 202 may be directed intoa single duct 208, where the air may be exposed to either heating orcooling temperature elements 210 integrated into the single duct 208,depending on the temperature requirement. Ambient air may bypass bothtemperature elements 210 integrated into the single duct 208. Otherembodiments may include heating or cooling the air flow into any numberof shutters 206, temperature ducts 208, and temperature elements 210,without departing from the scope of the invention.

In yet another reference to FIG. 5, step 8 516 describes the delivery ofair flow of variable flow rate and temperature-exposed air or ambientair onto the user. In an exemplary aspect, delivery oftemperature-treated or ambient air may be via dispensing nozzles 112 onthe front wall 126 of the haptic tower 408. The front wall 126 may havea dispensing nozzle 112 hidden behind a flush wall with the means ofopening and closing. The dispensing nozzle 112 may have separate outletsfor air, fog, and mist. Additionally, the dispensing nozzle 112 may havethe ability to rotate, or change the diameter of the inlet, in order totarget the direction of the air flow, as well as alter the intensity ofthe air flow. The haptic tower 408 may have a front wall 126 void ofdispensing nozzles 112, rather, the haptic delivery may be via a ventsystem, or any other outlet.

In further detail, still referring to step 8 516 of FIG. 5, the haptictower 408 may have a rotatable base 114, which may pivot the haptictower 408 in at least one axis of motion. A rotating base 114 allows forthe haptic tower 408 to rotate on its base to allow for more targeteddelivery of haptic effects. More particularly, a rotating base 114 mayallow for the haptic tower 408 to rotate on its base in at least oneaxis of motion to provide for a panning air flow effect. In otherembodiments, the rotatable base 114 may allow for motion along multipleaxis of rotation. In one embodiment, pivoting and targeted hapticdelivery may be further enhanced by using head tracking or full bodytracking system. Other embodiments may include a haptic tower 408 with adispensing nozzle 112, the pivoting and rotation of which may be alsoenhanced with the addition of head tracking or full body trackingsystems.

FIG. 6 and FIG. 7 are a system block diagram of the haptic engine in anexemplary environment according to an aspect of the invention. In anexemplary embodiment, a system for processing an audio and video inputin a point of view program for haptic delivery, comprises of, at leastone modular haptic tower 408. The haptic tower 408 further comprises of,at least one fan assembly, at least one duct, at least one outlet, aprocessor and a memory element coupled to the processor. Further yet, inanother preferred embodiment of the invention, a haptic engine 603,703comprises of an audio and video (a/v) buffer recognition block 604, 704;a haptic conversion block 605, 705 and a program executable by thehaptic engine 603, 703.

The haptic engine 603, 703 is further configured via a network 602, torecognize at least one of a data input from a user 601 and, or a virtualenvironment comprising the user 601, and determine for at least oneevent: any one of, or combination of, an event proximity score, a pixelcolor score of the event, a pixel velocity score of the event, and anaudio score of the event by the a/v buffer recognition block 604 704,apply a scoring rule for conversion of an at least one threshold-gradescored event into a haptic output command by the haptic conversion block605, 705.

Further yet, in an embodiment of the invention based on the hapticoutput command, the intensity of an actuator coupled to the at least onefan assembly and, or temperature element is controlled, resulting in avariable displacement and, or temperature of air through at least oneduct and at least one outlet of the modular haptic tower correspondingto the virtual environment 601 comprising the user 601. Alternatively,in an embodiment of the invention, the haptic output command controlsthe intensity of an actuator is by a haptic controller 606, 706 furthercontrolling the intensity of the fan assembly and, or the temperatureelement. In yet another embodiment of the invention, an odor recognitiontag may be further incorporated into the a/v recognition block to scorea smell sensation event.

Further yet, the network 602 may be any other type of network that iscapable of transmitting or receiving data to/from/between user devices:computers, personal devices, telephones or any other electronic devicesand user's audio-video environment. Moreover, the network 602 may be anysuitable wired network, wireless network, a combination of these or anyother conventional network, including any one of, or combination of aLAN or wireless LAN connection, an Internet connection, a point-to-pointconnection, or other network connection—either local, regional, orglobal. As such, the network 602 may be further configured with a hub,router, node, and, or gateway to serve as a transit point or bridge topass data between any of the at least networks. The network 602 mayinclude any software, hardware, or computer applications that implementa communication protocol (wide or short) or facilitate the exchange ofdata in any of the formats known in any art, at any time. In someembodiments, any one of a hub, router, node, and, or gateway mayadditionally be configured for receiving wearable or IoT data of amember/user of a group session, and such data may be saved, shared, orembedded within the session. Additionally, such personalized orcontextual data may further inform the suggestion tool layer orautomation tool layer on suggesting reactive or proactive routineswithin the workflow.

In a continuing reference to FIGS. 6 and 7, the network-coupled server602, cloud-based server, or haptic engine controller 603, 703 may be adevice capable of processing information received from at least one of,the user input 601 or user's surrounding audio-video environment 601.Other functionalities of the server or haptic engine 603, 703 mayinclude providing a data storage, computing, communicating andsearching. As shown in FIGS. 6 and 7, the server or haptic engine 603,703 processes the input, recognizes and scores the event, and convertsit into a haptic output command for further dynamic provisioning by thehaptic controller 606, 706.

Further yet, in an embodiment of the present invention, the data inputis from at least one of, device that outputs an audio and, or videosignal during operation. The audio, video outputs may be from any oneof, devices including, but not limited to, Closed-Circuit Television(CCTVs) cameras, High Definition (HD) cameras, non-HD cameras, handheldcameras, or any other video/image receiving units as well as the users'surrounding environments. The haptic engine 603, 703 may be configuredto receive a dynamic imagery, audio or video footage from theaudio/video receiving devices, and transmit the associated data to thea/v recognition block 604, 704 for further dynamic provisioning. In anembodiment, the memory element coupled to the processor may maintain thedynamic audio/video footage as received from the video/image receivingdevices. Alternatively, the audio/video inputs may be archived andstored in data storage element coupled to a processor that is configuredto store pre-recorded or archived audios/videos. The audio/video inputsmay be stored in any suitable formats as known in the art or developedlater. The audio/video input archive may include a plurality of localdatabases or remote databases. The databases may be centralized and/ordistributed. In an alternate scenario, the audio/video input archivesmay store data using a cloud based scheme.

Now with reference to FIG. 8, in an embodiment of the invention, thehaptic engine 803 comprises of at least one of, an a/v recognition block804, a haptic conversion block 805 and a haptic controller 806. Thehaptic engine 803 may be further configured to recognize at least oneof, a data input from a user and, or a virtual environment comprisingthe user. Further yet, the a/v recognition block 804 tags at least oneevent 804 a for scoring by at least one of, or a combination of, motion,color and, or sound.

Further yet, the a/v recognition block 804 determines a proximity score804 b of the tagged event 804 a by determining the distance from any oneof, a selected target zone comprising the event and, or a selecteddestination zone comprising the user, within a matrix of zones thatoccupy the entire field of view and, or sound.

In yet another embodiment of the invention, the a/v recognition block804 determines a pixel color score 804 c of the tagged event 804 a bycalculating an average hue score of the tagged event 804 a using pixeldata in a screen buffer by calculating a nearness coefficient,calculating an average of red & blue channels in the screen buffer,calculating an offset coefficient, calculating an average luminance inthe screen buffer and deriving the average pixel score of the taggedevent 804 a based on an aggregation of the coefficients. Additionally,in an embodiment of the invention, the a/v recognition block 804determines a pixel velocity score 804 d of the tagged event 804 a basedon the coefficient by capturing a series of frames, and calculates acoefficient related to pixel velocity 804 d by testing the per-frame andper-range delta in any one of, or combination of hue, luminance,brightness, saturation and, or color value.

Further yet, in an embodiment of the invention, the a/v recognitionblock 804 determines an audio score 804 e of the tagged event 804 abased on a coefficient by capturing an audio buffer and calculating anAverage Energy, Immediate Energy, Immediate Energy Delta & ImmediateEnergy Mean Deviation and further, calculating a coefficient related tobroad and narrow changes in a frequency spectrum.

Further yet, in an embodiment of the invention, the haptic conversionblock 805 of the haptic engine 803 applies a scoring rule 805 a for theconversion of at least one threshold-grade scored event into a hapticoutput command 805 b. Further yet, the haptic a/v conversion block 804is further coupled to a haptic conversion block 805 and the hapticcontroller 806 which further, processes the haptic output command 805 bfor actuating a fan and, or temperature element disposed within themodular haptic tower. Finally, based on the haptic output command 805 b,the haptic controller 806 may control an intensity 806 b of an actuator806 a coupled to the at least one fan assembly and, or temperatureelement, resulting in a variable displacement and, or temperature of airthrough the at least one duct and at least one outlet of the modularhaptic tower corresponding to the virtual environment comprising theuser.

In yet another embodiment of the invention, the haptic engine system 803may comprise a feed-forward and, or back-propagated neural networktrained to trigger a haptic output 805 b based on any one of, orcombination of, a stored data input, stored tagged event 804 a, storedcoefficient value, stored event proximity score value 804 b, storedpixel color score value 804 c, stored pixel velocity score value 804 d,stored audio score value 804 e, and, or haptic output command 805 b. Forexample, consider a scenario of a campfire, wherein the haptic outputcommands 805 b configured by the system are based on any one of, or acombination of, but not limited to, heat, crackling sound, windvelocity, burning sensation, sudden impact. If the tagged event in avirtual environment proximal to the user is of a heavily burningcampfire, then the a/v recognition block 804 will generate a unique tagfor an event 804 a, compute a pixel proximity score 804 b, pixel colorscore 804 c, pixel velocity score 804 d, and an audio score, whichcorresponds to a series of haptic outputs commands comprising of aburning sensation, hot air and a crackling sound.

Further yet, if the campfire is under control and, or if the user movesfarther away from the site, or if it would start to rain, then the a/vrecognition block 804 will generate another unique tag for an event 804a, compute another pixel proximity score 804 b, pixel color score 804 c,pixel velocity score 804 d, and, or an audio score 804 e, which maycorresponds to a series of another set of haptic outputs commands 805 bthus, comprising a less burning sensation, warm air and a faintercrackling sound. Furthermore, as the user in the virtual environmentcontinues to move farther away from the campfire or if it would start torain heavily, the burning campfire event may eventually be scored acrossall parameters below a predefined threshold, thereby no longercommanding any one of a haptic effect commands. In an alternativeembodiment of the invention, an odor recognition tag may be incorporatedinto the a/v recognition block 804 to score an odor haptic output 805 b.

Additionally, in another embodiment of the invention, the system, maycomprise a feed-forward and, or back-propagated neural network to use aseries of externally captured buffers containing known audio-visualsources to aid in real-time recognition of the audio and video input byusing a probabilistic approach to determine presence in a capturedbuffer. The audio/video input events may be tracked in a current frameand stored in a computer processor database for machine learningobjectives. A classification algorithm may be based on supervisedmachine learning techniques such-as SVM, Decision Tree, Neural Net, AdaBoost, and the like. Further, the classification may be performed byanalyzing one or more features based on any one of, or combination of, astored data input, stored tagged event 804 a, stored coefficient value,stored event proximity score value 804 b, stored pixel color score value804 c, stored pixel velocity score value 804 d, stored audio score value804 e, and, or haptic output command 805 b.

In another embodiment of the present invention, the classificationalgorithm may employ an unsupervised machine learning to learn thefeatures from the image input data itself. For example, a Neural NetworkAutoencoder can be used to learn the features and then to train a DeepNeural Network or a Convolutional Neural Network. The classificationalgorithm may be based on a supervised or an unsupervised machinelearning technique, and the classification is performed by analyzing oneor more features of the tracked objects. Examples of the one or morefeatures include, but are not limited to, a size, an aspect ratio, alocation in the scene, and other generic features such as color, HoG,SIFT, Haar, LBP, and the like. Typically, the object classificationalgorithm is executed on top of object tracking algorithm and it allowsto localize search region, thus decreasing the amount of computation.Such approach results in reducing power consumption and/or increase thedetection speed and accuracy.

FIGS. 9 and 10 shows the overall interaction flow of the haptic engine,according to an embodiment of the present invention. Both figuresillustrate a system for processing an audio and video input 910, 1010 ina point of view program for haptic delivery, said system comprising: amodular haptic tower; a processor; a memory element coupled to theprocessor; a haptic engine comprising: an audio and video (a/v) bufferrecognition block; a program executable by the haptic engine andconfigured to: recognize a data input from any one of, or both, a userand a virtual environment comprising the user, and determine for atleast one event: any one of, or combination of, an event proximityscore, a pixel color score of the event, a pixel velocity score of theevent, and an audio score of the event 940 by the a/v buffer recognitionblock; and convert the at least one scored event into a haptic outputcommand 950 and based on the haptic output command 950, control anintensity of an actuator 960, 1060 resulting in a variable displacementof any one of, or combination of, air, temperature, mist, pressure and,or impact corresponding to the virtual environment comprising the user1080.

The various haptic effects commanded by the haptic output command 950may be any one of, or combination of wind/speed 1080 a, heat/cool 1080b, sudden impact 1080 c, water effects 1080 d, and, or strike/pressure1080 e. For instance, if the event proximal to the user is a heavyflowing, cold, water fall, then the haptic engine will compute a pixelproximity score, color score, velocity score, and audio score, whichcorresponds to a series of haptic outputs comprising a strong burst ofcold air, followed by a heavy spray of cold water—simulating a heavywind 108 a and a heavy mist 1080 d. Conversely, if the same heavyflowing and cold water fall is not proximal to the user, the ensuingpixel color score, velocity score, and audio score may correspond to aseries of haptic outputs 950 comprising of just a light air flow from afan assembly 960, 1060 and a light water spray from the water spray unit960, 1060—simulating a light wind 1080 a and a light mist 1080 d. As theuser in the virtual environment is walking away from the water fallevent, and it is distancing in the frame, the scores will be reflected,leading to a winding down of actuator intensity 960 and haptic effect.As the user in the virtual environment continues to walk away, the waterfall event may eventually be scored across all parameters below apredefined threshold, thereby no longer commanding any one of a hapticeffect 1080.

While not shown in FIG. 9 or 10, the haptic engine may be coupled to afeed-forward and, or back-propagated neural network trained to trigger ahaptic output based on any one of, or combination of, a stored datainput, stored tagged event, stored coefficient value, stored eventproximity score value, stored pixel color score value, stored pixelvelocity score value, stored audio score value, and, or haptic outputcommand. The feed-forward or back-propagated neural network may furtheruse a series of externally captured buffers containing knownaudio-visual sources to aid in real-time recognition of the audio andvideo input by using a probabilistic approach to determine presence in acaptured buffer.

For instance, when the user walks away from the water fall event in theprevious scenario and walks toward another scenario featuring an eventincluding rushing water, such as white-water rafting, the engine may usethe machine learning techniques to use at least one of the scoringvalues of the white-water rafting event to predict the other scoringvalues based on the similarities of the first scoring values with theearlier stored waterfall event. This predictive scoring may ensurequicker haptic output response time, in addition to reducing computingresources. Likewise, the machine-learning coupled system needs to bediscriminative enough to avoid false positives. For instance, if theuser walks away from the water-fall event and soon stumbles upon afast-flowing creek (another event featuring rushing water), it needs tobe able to discriminate between this and a rushing white-water raftingscenario. Despite the fact that perhaps all three water featuring eventsmay score for pixel color similarly, they may each have varying pixelvelocity scores, thereby commanding for varying wind intensities. Insuch a scenario, the system may have to root through the larger cache ofsimilar events and do a deeper stage calculation of each paired ormatched event. In such scenarios, wherein multiple cached events may beimplicated due to their similarity, the system may require atwo-parameter check-point in order to trigger predictive scoring valuesand a haptic command output.

In one embodiment, the machine learning systems may differentiatebetween background events and foreground events dynamically. Inputframes may correspond to complex scenes that include areas withsignificant background variations/continuous movements. However, thesevariations and continuous movements should not trigger a hapticexpression since they are background events, and not foreground events,such as flying birds, swaying tree branches, moving clouds, etc. Thesystem, referencing a background event cache, can label the event as abackground event, bypassing the need for the a/v recognition block totag and compute the event (event proximity score). Furthermore, based onthis background event referencing, events may be labeled as background,even if they appear in the foreground and score a threshold-gradeproximity score. For instance, a moving cloud passing over a rushingwaterfall should not interfere with the haptic expression profile of therushing waterfall, despite the fact that the moving cloud may impair thepixel color score of the rushing waterfall. The moving cloud would bedetected as a background event based on background event cachereferencing, and subsequently, the final pixel color score of therushing waterfall would account for the moving cloud. The backgroundevent cache may further differentiate between static background eventsand dynamic background events. In an embodiment, a different algorithm/smay be applied for depending on the background event be labeled asstatic or dynamic.

In an embodiment, once the background event is extracted out, thenremaining events in the input frame may be referenced from a foregroundevent cache, once at least one parameter triggers the event. As with thebackground event referencing, an algorithm/s may be applied for acurrent triggered event in the input frame, and foreground event binswith similar event/score features as the current event are identified.Event triggering and haptic output expression based on the a/vrecognition block or machine learning may be based on thresholdcalculations employing a Local Adaptive Thresholds (LAT) technique andthe threshold may be adapted dynamically.

In one embodiment, the cache of events or scores corresponding to eventsare constantly updated over time. For example, the cache is updated tohandle gradual time variations (such as night/day changes), and aplurality of other background events (moving clouds/sun, shadows,weather, etc.). Moreover, the cache update may also involve spatialchanges (neighboring pixel changes) within the input frames. To thisend, background changes can be accounted for by the system using theselearned approaches and not affect the haptic expressions of the targetedforeground events. For instance, the rushing waterfall should translatefor a similar haptic expression or profile, irrespective of changes inlighting or color due to variations in time of day, weather, or cloudcoverage, etc.

In other embodiments, in addition to classifying events as background orforeground, events may be further classified in terms of category, suchas animal, human, projectile, natural phenomenon, crafts, vehicles, etc.Categorization may be based on at least one visual and, or audio aspect:color, size, aspect ratio, etc. In another embodiment of the presentinvention, the categorization algorithm categorizes the event,supervised by machine learning, and then inserts into categorized binswithin either the background event cache or foreground event cache.Furthermore, machine learning approaches, such as a Deep Neural Networkor a Convolutional Neural Network, may match a live event feature orscore parameter to a cached event in any one of a category event binwithin the background event cache or foreground event cache.

FIG. 11 is a process flow diagram illustrating the steps involved from adata input to a haptic output, as the commands are passed down throughthe haptic engine. The a/v recognition block tags at least one event forscoring by displaying any one of, or combination of, motion, color, orsound 1110. If at least one of these characteristics are identifiedabove a threshold, then the a/v recognition module tags the event forscoring 1120.

The a/v recognition block determines a proximity score of the taggedevent by determining distance from any one a selected target zonecomprising the event and a selected destination zone comprising theuser, within a matrix of zones occupying the entire field of view and,or sound 1130. Once the tagged event is determined as proximal over athreshold, then a/v recognition block determines a pixel color score ofthe tagged event 1140 by calculating an average hue score of the taggedevent using pixel data in a screen buffer, and calculate a nearnesscoefficient; calculate an average of red & blue channels in the screenbuffer, and calculate an offset coefficient; calculate an averageluminance in the screen buffer; and deriving the average pixel score ofthe tagged event based on an aggregation of the coefficients.

Simultaneously, the tagged proximal event is also processed by the a/vrecognition block, which may determine a pixel velocity score of thetagged event 1150 by capturing a series of frames, and calculate acoefficient related to pixel velocity by testing the per-frame andper-range delta in any one of, or combination of hue, luminance,brightness, saturation, and, or color value; and deriving the pixelvelocity score of the tagged event based on the coefficient.

Simultaneously, the a/v recognition block determines an audio score ofthe tagged event 1160 by capturing an audio buffer and calculate anAverage Energy, Immediate Energy, Immediate Energy Delta, and ImmediateEnergy Mean Deviation, and calculate a coefficient related to broad andnarrow changes in a frequency spectrum; and deriving the audio score ofthe tagged event based on the coefficient.

Upon scoring of any one of, or combination of, the video and audioaspects of the tagged event, the scored-tagged events may be referencedagainst cached/binned scores/events to translate into a haptic outputcommand. In some embodiments, the scored-tagged events may input into ahaptic conversion block, applying a scoring rule 1170, wherein any of atagged and scored event is a threshold-grade scored event, and saidthreshold-grade scored event is converted into a haptic output commandby the haptic conversion block. In other embodiments, a scoring rule orthreshold calculation techniques, such as Local Adaptive Threshold (LAT)may be used to determine whether the scored event is in factthreshold-grade and warranting a haptic output command for hapticexpression.

The haptic conversion block may be further coupled to a hapticcontroller, and said haptic controller processes the haptic outputcommand for actuating any one of, or combination of, a fan, temperatureelement, displacement chamber, water mist unit, aroma unit, tactilemember, and, or tactile projectile unit, disposed within the modularhaptic tower. Alternatively, a series of haptic effects may beachievable employing the haptic engine, wherein the haptic effects arenot disposed within the modular haptic tower. For instance, the hapticeffects may be disposed within a haptic vest, glove, or any otherwearable, and configured to actuate based on the audio-video inputprocessed by the haptic engine.

Furthermore, the system may engage in processing shortcuts by employinga feed-forward and, or back-propagated neural network trained to triggera haptic output based on any one of, or combination of, a stored datainput, stored tagged event, stored coefficient value, stored eventproximity score value, stored pixel color score value, stored pixelvelocity score value, stored audio score value, and, or haptic outputcommand. The system may reference a live tagged event to a cached orbinned event by at least one point of event feature or score matching,and shotgun a haptic output command and, or haptic output expression.Furthermore, the feed-forward and, or back-propagated neural network mayuse a series of externally captured buffers containing knownaudio-visual sources to aid in real-time recognition of the audio andvideo input by using a probabilistic approach to determine presence in acaptured buffer.

In yet other embodiments, a reiterative module may be further comprisedin the haptic engine, wherein the reiterative module links andcontinuously reiterates the currently played haptic output, despite thehaptic provoking event being out of the frame. For instance, even whenthe haptic provoking event is out of the frame and no longer registeringa pixel color score, pixel velocity score, or audio score, thereiterative module may persist the haptic command and output, providedthe pixel proximity score remains within the acceptable threshold. Inkeeping with our rushing waterfall scenario, after provoking the hapticexpression for the rushing waterfall, the haptic expression may persist,despite the user turning around, and the waterfall no longer being inthe frame. Once the user walks away by a threshold-dependent distance,the haptic expression corresponding to the rushing waterfall maycease—with or without the supervision of the reiterative module ormachine learning.

FIG. 12 illustrates a method for processing an audio and video input ina point of view program for haptic delivery, said method comprising thesteps of: (step 1) recognizing any one of a data input from any one of auser and a virtual environment comprising the user, and determine for atleast one event: any one of, or combination of, an event proximityscore, a pixel color score of the event, a pixel velocity score of theevent, and an audio score of the event by an a/v buffer recognitionblock 1210; and (step 2) converting the at least one scored event into ahaptic output command and based on the haptic output command, control anintensity of an actuator coupled to at least one fan assembly and, ortemperature element, resulting in a variable displacement and, ortemperature of air through the at least one duct and the at least oneoutlet corresponding to the virtual environment comprising the user1220.

As shown in FIG. 13 (system block diagram of the peripheral modulationof the unscripted feed in accordance with an aspect of the invention),the processor 1310 may further comprise an engine 1320 (alternatively, ahaptic engine) to be able to recognize at least one of the audio orvideo input from the at least one of the original programming feed orlive feed, and determine for at least one tagged event, at least one ofa pixel color score 1320 a, a pixel velocity score 1320 b, an eventproximity score 1320 c or an audio score 1320 d. In a preferredembodiment, more than one score event will be accounted for to convertinto an output command.

Once the scored events are tabulated, the processor 1310 (hapticconversion 1330/output 1330 a) may convert the at least one scored eventinto at least one of an output command that triggers or controls amodulation effect of the at least one peripheral device in physicalcontact or free from the user in communication with the at least thefirst device playing the at least one of the original programming feedor live feed, thereby enabling modulation (controlled by the modulator1340) of the at least one peripheral device based on any programmingcomprising at least one of an audio or video input and not requiringscripted modulation triggers.

In an embodiment (not shown), the processor may comprise of at least oneof an a/v recognition block (engine 1320), a haptic conversion block(output 1330 a) and a haptic controller (modulator 1340). The processor1310 (engine 1320/recognition block) determines a proximity score 1320 cof the tagged event by determining the distance from any one of, aselected target zone comprising the event and, or a selected destinationzone comprising the user, within a matrix of zones that occupy theentire field of view and, or sound. The engine 1320 may furtherdetermine a pixel color score 1320 a of the tagged event by calculatingan average hue score of the tagged event using pixel data in a screenbuffer by calculating a nearness coefficient, calculating an average ofcolor channels in the screen buffer, calculating an offset coefficient,calculating an average luminance in the screen buffer and deriving theaverage pixel score of the tagged event based on an aggregation of thecoefficients. Pixel velocity scores 1320 b of the tagged event arecalculated by the engine 1320 based on the coefficient by capturing aseries of frames, and calculating a coefficient related to pixelvelocity by testing the per-frame and per-range delta in any one of, orcombination of, hue, luminance, brightness, saturation and, or colorvalue. Further yet, the engine 1320 determines an audio score 1320 d ofthe tagged event based on a coefficient by capturing an audio buffer andcalculating an Average Energy, Immediate Energy, Immediate Energy Delta& Immediate Energy Mean Deviation and further, calculating a coefficientrelated to broad and narrow changes in a frequency spectrum.

Further yet, in an embodiment of the invention, the haptic conversionblock (conversion block 1330/output 1330 b) may apply a scoring rule forthe conversion of at least one threshold-grade scored event into ahaptic/output command. Further yet, the haptic/conversion block 1330 isfurther coupled to a haptic controller/modulator 1340, which further,processes the haptic output command for actuating modulating anyperipheral device capable of modulation whether it be in physicalcontact or free the at least one user. In one embodiment, thehaptic/output 1330 b, the haptic controller/modulator 1340 may control aswitch/intensity of an actuator coupled to a motor output or any otherarticulation/mechanized operation inherent in a peripheral device. Forinstance, the haptic controller/modulator 1340 may control aswitch/intensity coupled to a motor output coupled to the at least onefan assembly and, or temperature element, resulting in a variabledisplacement and, or temperature of air through the at least one ductand at least one outlet of the modular haptic tower corresponding to thevirtual environment comprising the user.

While not shown in FIG. 13, the processor/haptic/engine 1320 maycomprise a feed-forward and, or back-propagated neural network trainedto trigger an output 1330 b based on any one of, or combination of, astored data input, stored tagged event, stored coefficient value, storedevent proximity score value, stored pixel color score value, storedpixel velocity score value, stored audio score value, and, or storedoutput command. For example, consider a scenario of an erupting volcano,wherein the output 1330 b configured by the system are based on any oneof, or a combination of, but not limited to, heat, erupting sound,flowing sound, wind velocity, burning sensation, sudden impact. If thetagged event in a programming environment proximal to the user is of aheavily erupting volcano, then the a/v recognition block/engine 1320will generate a unique tag for an event, compute a pixel proximity score1320 c, pixel color score 1320 a, pixel velocity score 1320 b, and anaudio score 1320 d, which corresponds to a series of output commandscomprising of a light effect, burning sensation, hot air, wind effects,crackling sound, eruption sound, flowing sound, and rumbling modulatedfrom any one of a peripheral device: Interactive chair, home integratedspeakers, controller devices, heat lamp, fan, modular haptic tower, homeintegrated light sources, gloves, vest, etc.

Further yet, if the volcano eruption is under control and, or if theuser moves farther away from the site, or if it would start to rain,then the a/v recognition block/engine 1320 will generate the countereffect with low latency: Another unique tag for the event, computeanother pixel proximity score 1320 c, pixel color score 1320 a, pixelvelocity score 1320 b, and, or an audio score 1320 d, which maycorresponds to a series of another set of output commands 1330 b thus,comprising a less burning sensation, less warm air and a faintereruption, flowing, or crackling sound. Furthermore, as the userexperiences a character from the program moving farther away from thevolcano or if it would start to rain heavily, the erupting volcano eventmay eventually be scored across all parameters below a predefinedthreshold, thereby no longer commanding any one of the modulatingeffects. In an alternative embodiment of the invention, an odorrecognition tag may be incorporated into the a/v recognitionblock/engine 1320 to score an odor haptic output from an odor dispersingdevice. Also, alternatively, latency may improved between effect andcounter effect by simply commanding an inverse voltage applied to amotor output of any of the relevant peripheral devices to effectuate thecounter effect with a quicker response time and reducing the latencybetween the effect-counter effect user experience.

Additionally (also not shown), in another embodiment of the invention,the system, may comprise a feed-forward and, or back-propagated neuralnetwork to use a series of externally captured buffers containing knownaudio-visual sources to aid in real-time recognition of the audio andvideo input by using a probabilistic approach to determine presence in acaptured buffer. The audio/video input events may be tracked in acurrent frame and stored in a computer processor database for machinelearning objectives. A classification algorithm may be based onsupervised machine learning techniques such-as SVM, Decision Tree,Neural Net, Ada Boost, and the like. Further, the classification may beperformed by analyzing one or more features based on any one of, orcombination of, a stored data input, stored tagged event, storedcoefficient value, stored event proximity score value, stored pixelcolor score value, stored pixel velocity score value, stored audio scorevalue, and, or haptic output command 1330 b.

In another embodiment of the present invention, the classificationalgorithm may employ an unsupervised machine learning to learn thefeatures from the image input data itself. For example, a Neural NetworkAutoencoder can be used to learn the features and then to train a DeepNeural Network or a Convolutional Neural Network. The classificationalgorithm may be based on a supervised or an unsupervised machinelearning technique, and the classification is performed by analyzing oneor more features of the tracked objects. Examples of the one or morefeatures include, but are not limited to, a size, an aspect ratio, alocation in the scene, and other generic features such as color, HoG,SIFT, Haar, LBP, and the like. Typically, the object classificationalgorithm is executed on top of object tracking algorithm and it allowsto localize search region, thus decreasing the amount of computation.Such approach results in reducing power consumption and/or increase thedetection speed and accuracy.

In one embodiment, the machine learning systems may differentiatebetween background events and foreground events dynamically. Inputframes may correspond to complex scenes that include areas withsignificant background variations/continuous movements. However, thesevariations and continuous movements should not trigger a modulationsince they are background events, and not foreground events, such asflying birds, swaying tree branches, moving clouds, etc. The system,referencing a background event cache, can label the event as abackground event, bypassing the need for the a/v recognition block totag and compute the event (event proximity score). Furthermore, based onthis background event referencing, events may be labeled as background,even if they appear in the foreground and score a threshold-gradeproximity score. For instance, a moving cloud passing over a rushingwaterfall should not interfere with the haptic expression profile of therushing waterfall, despite the fact that the moving cloud may impair thepixel color score of the rushing waterfall. The moving cloud would bedetected as a background event based on background event cachereferencing, and subsequently, the final pixel color score of therushing waterfall would account for the moving cloud. The backgroundevent cache may further differentiate between static background eventsand dynamic background events. In an embodiment, a different algorithm/smay be applied for depending on the background event be labeled asstatic or dynamic.

In an embodiment, once the background event is extracted out, thenremaining events in the input frame may be referenced from a foregroundevent cache, once at least one parameter triggers the event. As with thebackground event referencing, an algorithm/s may be applied for acurrent triggered event in the input frame, and foreground event binswith similar event/score features as the current event are identified.Event triggering and haptic output expression based on the a/vrecognition block or machine learning may be based on thresholdcalculations employing a Local Adaptive Thresholds (LAT) technique andthe threshold may be adapted dynamically.

In one embodiment, the cache of events or scores corresponding to eventsare constantly updated over time. For example, the cache is updated tohandle gradual time variations (such as night/day changes), and aplurality of other background events (moving clouds/sun, shadows,weather, etc.). Moreover, the cache update may also involve spatialchanges (neighboring pixel changes) within the input frames. To thisend, background changes can be accounted for by the system using theselearned approaches and not affect the haptic expressions of the targetedforeground events. For instance, the rushing waterfall should translatefor a similar haptic expression or profile, irrespective of changes inlighting or color due to variations in time of day, weather, or cloudcoverage, etc.

In other embodiments, in addition to classifying events as background orforeground, events may be further classified in terms of category, suchas animal, human, projectile, natural phenomenon, crafts, vehicles, etc.Categorization may be based on at least one visual and, or audio aspect:color, size, aspect ratio, etc. In another embodiment of the presentinvention, the categorization algorithm categorizes the event,supervised by machine learning, and then inserts into categorized binswithin either the background event cache or foreground event cache.Furthermore, machine learning approaches, such as a Deep Neural Networkor a Convolutional Neural Network, may match a live event feature orscore parameter to a cached event in any one of a category event binwithin the background event cache or foreground event cache.

Now in reference to FIG. 14. FIG. 14 shows the overall interaction flowof the recognition engine 1420, output 1430, and peripheral devicecontrols 1440, in accordance with an embodiment of the presentinvention. The figure illustrates a system for processing an audio andvideo input from an unscripted programming feed (original programmingand, or live feed) for modulating a peripheral device (in physicalcontact or free from at least one user), the system comprising: at leasta first device and at least one peripheral device in short-range ornetworked communication with one another, wherein the engine 1420 isconfigured to receiving at least one of a video or audio signal from atleast one of an original programming feed or live feed; and generating atriggering signal in response to the at least one of the video or audiosignal from the at least one of the original programming source or livefeed that match or exceed a threshold score for a scored event andtriggering or controlling at least one modulation of the at least oneperipheral device by the output 1430, thereby enabling modulation of theat least one peripheral device 1440 corresponding to at least one of theprogramming feed or live feed not scripted with a modulation trigger.

While not shown in FIG. 14, the at least first device may be coupled toa display screen for user viewing and the at least first device may bein communication to the at least one peripheral device in physicalcontact with the at least one user or free from the at least one userfor triggering at least one of a modulation (actuation or haptic effect)1440. In a preferred embodiment, the at least one peripheral device is adevice for controlling viewing operation of the at least one of theoriginal programming feed or live feed displayed on the screen coupledto the at least first device. At least one of the audio or video inputfrom at least the first device that outputs an audio and, or videosignal during operation is in its original programming feed form or livefeed form for triggering or controlling at least one of a modulatingeffect (actuation or haptic effect) from a peripheral device 1440 inphysical contact with the at least one user or free from the at leastone user, wherein the at least first device and peripheral device arethe same device. For instance, the first device and peripheral device1440 may be the same mobile phone or tablet giving off a tactilefeedback, sound feedback, or light feedback in response to a programmingevent, without having triggering cues embedded in the programmingcorresponding to said feedback.

Alternatively, the system may further comprise a plurality of peripheraldevices (similar or heterogenous) 1440 with at least one in physicalcontact with the at least one user or free from the at least one userand in communication to the same original programming feed or live feedfrom the at least first device. The plurality of peripheral devices 1440may be modulated to disperse a synergistic effect or heterogenous effectdelivering an enhanced immersive experience corresponding to theprogramming.

In one embodiment, the at least first device is playing a programmingfeed comprising audio signals with a frequency imperceptible to a humanear (sub-audio), whereby the sub-audio signal triggers or controls themodulation effect from the at least one peripheral device in physicalcontact with the user or free from the user.

Preferably, the at least first device is at least one of a computing,gaming, streaming, television, or audio or video playback device playingan original programming feed and the at least one peripheral device 1440in physical contact with the at least one user is at least one of ahaptic triggering glove, thimble, vest, jacket, wearable, watch, mobilephone, tablet, joystick, toy, erotic toy, game controller, interactiveseat, head phones, or head gear. Modulating effects may range fromtactile feedback, sound feedback, light feedback, air feedback, motionfeedback, temperature feedback, olfactory feedback, etc. In anotherembodiment, the at least first device is at least one of a computing,gaming, streaming, television, or audio or video playback device playingan original programming feed and the peripheral device 1440 is free fromthe user and is at least one of a stand-alone haptic tower, heat lamp,fan, light source, light fixture, house alarm, or IoT hub. Modulatingeffects may range from tactile feedback, sound feedback, light feedback,air feedback, motion feedback, temperature feedback, olfactory feedback,etc.

In other embodiments, the at least first device is at least one of acamera, microphone, sensor, or audio or video capture for playing a livefeed and the peripheral device 1440 in physical contact with the user isat least one of a mobile phone, biomedical tool, erotic toy, steeringwheel, or automobile pedal. Modulating effects similarly range fromtactile feedback, sound feedback, light feedback, air feedback, motionfeedback, temperature feedback, olfactory feedback, etc. In yet otherembodiments, the at least first device is at least one of a camera,microphone, sensor, or audio or video capture for playing a live feedand the peripheral device 1440 free from the user is at least one of anautomobile alarm, house alarm, stand-alone haptic tower, heat lamp, fan,light source, light fixture, thermostat, or IoT hub. Modulating effectsalso similarly range from tactile feedback, sound feedback, lightfeedback, air feedback, motion feedback, temperature feedback, olfactoryfeedback, etc.

For instance, the at least one peripheral device 1440 may be an eroticdevice intended for sexual pleasure for at least one of a male or femalecomprising at least one of a sleeve-lined tube or phallic-shaped memberwith modulation to mimic at least one of a sexual act displayed from theprogramming played on the at least first device. For example, a user mayengage the erotic device and experience the same pleasure experienced bythe sex-engaged character from the programming in real-time. Therefore,the erotic device feedback mirroring sex-engaged characters is notlimited to a trigger-embedded library of content, but rather, may beplug-n-played with any sex-driven programming. Alternatively, eroticdevices may be in communication with each other and be engaged by remoteusers and receiving corresponding feedback in real-time.

In another example, the at least one peripheral device 1440 is at leastone of a removable or fixed fixture of a light source with modulation tocorrespond to at least one of a display or audio of programming playedon the at least first device. This may apply to home use or a largervenue setting with a congregation of people experiencing the sameprogramming from the same first device (speakers, display wall, or liveevent). For instance, the lighting system and display system may be incoordination with the sound system without modulating triggers beingembedded in the audio input. Therefore, the lighting and display for theclub may plug-n-play with any audio output, without being restricted toa trigger-embedded library of content.

In yet another example, the at least one peripheral device 1440 is amobile phone with modulation to correspond to at least one of a displayor audio of programming played on the at least first device; and whereinthe at least one peripheral device and the at least one first device arethe same device. For instance, a user's mobile phone may vibrate everytime a user's favorite team scores a goal while live streaming a soccermatch. In this scenario, the user may configure output parameters toinstruct the engine/system 1420 to drive the tactile/vibrationalfeedback strictly upon a user-selected team scores. In another scenario,output parameters may instruct the engine 1420 to drive thetactile/vibrational feedback upon a user-selected team scoring a goal,and drive a lighting/sound feedback upon a user-selected player scoringa goal.

In another embodiment, the at least one peripheral device is aninteractive seat or chair with modulation to correspond to at least oneof a display or audio of programming played on the at least firstdevice. The interactive seat may be intended for home use or as part ofa collection in a venue, such as a movie theater, concert hall, stadium,etc. The chair may vibrate, rock, pitch, yaw, roll, etc. as expressionsof modulation in response to the system event recognition/scoring1420/output 1430. In other words, the interactive seat/chair may providemotion and tactile/vibrational feedback corresponding to anyprogramming, and not just to a library of content curated with embeddedmodulation triggers.

Audio input (sub-audio or supra-audio) may be visualized using computervision aspects by converting the audio into a visual representationusing FFT and STFT. Transforming a time-based signal into afrequency-based signal using the fast fourier transform (FFT) orshort-time fourier transform. (STFT). Alternatively, a predefined bufferor ‘wavelet’ (snapshot of a waveform) may be used to search for thepresence of a given frequency or signature in a time-based signal also.One may also look at the first and second derivatives of the individualfrequencies. Instantaneous change, and rate of change. For allcalculated coefficients, we calculate the 1st derivative, 2ndderivative, and sometimes 3rd derivatives. Computer vision may then lookfor patterns and feeding that data into machine learning to identifyspecific failure cases that may have non-linearities. For example, afailing ball bearing will have an audio pattern that will change overtime. For industrial applications, you could listen to an oil pump or afracking drill and be able to determine material hardness for rock youare drilling, or if you have a clog for when oil is being pumped througha pipe. As another example, FFT and STFT could convert an audio inputinto a visual representation for purposes to diagnose a mechanical orphysiological ailment, in addition to driving a modulation of aperipheral device.

In reference to FIG. 15, which depicts a method for processing at leastone of an audio or video input for non-scripted modulation of at leastone peripheral device, the method comprises the steps of: (1)recognizing at least one of the audio or video input from the at leastone of the original programming feed or live feed, and determining forat least one tagged event, at least one of a pixel color score, a pixelvelocity score, an event proximity score or an audio score 1520; and (2)converting the at least one scored event into at least one of an outputcommand that triggers or controls a modulation effect of the at leastone peripheral device in physical contact or free from the user incommunication with the at least the first device playing the at leastone of the original programming feed or live feed, thereby enablingmodulation of the at least one peripheral device based on anyprogramming comprising at least one of an audio or video input and notrequiring scripted modulation triggers 1530.

In summation, modulation effects of peripheral devices are not triggeredby embedding triggering cues via a developer kit or after-market coding(scripted programming feed), but rather, directly integrative to theoriginal programming feed or live feed in a plug-n-play fashion viacomputer vision processing (unscripted programming feed)—therebyobviating content hurdles and opening the full library of a/v basedprogramming in communication with a peripheral device. Content no longerneeds to be limited to within provider and developer silos in order tobe coupled to a fully immersive experience.

Now in reference to FIG. 16, which depicts a method flow diagram of thelight-emitting peripheral modulation of the unscripted feed inaccordance with an aspect of the invention. In a preferred embodiment, amethod for processing at least one of an audio or video input fornon-scripted light modulation of at least one light-emitting peripheraldevice (LEPD) comprises the steps of: (1) recognizing at least one ofthe audio or video input from at least one first device (D1) anddetermine for at least one tagged event, at least one of a pixel colorscore, a pixel velocity score, an event proximity score or an audioscore 1620; and finally (2) converting the at least one scored eventinto at least one of an output command that triggers or controls alight-emitting effect of the at least one LEPD, whereby the LEPD isfurther at least one of a mouse, keyboard, headset, speaker, joystick,D1-coupled display, television monitor, tablet, smart phone, room lightsource, or home IoT (Internet-of-Things) hub, and whereby the DI is atleast one of a gaming console, desktop, laptop, tablet, smartphone,playback device, television monitor, display, or home IoT hub 1640.

In other embodiments, the method may entail a first step of: recognizingat least one of the audio or video input from at least one first device(D1) and determine for at least one tagged event, at least one of apixel color score, a pixel velocity score, an event proximity score oran audio score; and the last step of commanding a trigger or controlover a light-emitting effect of the at least one light-emittingperipheral device (LEPD) upon a threshold-grade score. Examples of anLEPD may be at least one of a mouse, keyboard, headset, speaker,joystick, D1-coupled display, television monitor, tablet, smart phone,room light source, or home IoT (Internet-of-Things) hub. Examples of aD1 may be a gaming console, desktop, laptop, tablet, smartphone,playback device, television monitor, display, or home IoT hub. The LEPDand D1 may be in communication with one another using any number ofshort-range wireless or wired communication. Communication between theLEPD and D1 may additionally be achieved with a transit hub, such as arouter, home automation hub, display, or any other communicationinterface. While LED's are a preferred light-emitting source, othersources may be disposed on any one of LEPD's, such as CFL's, Halogens,Incandescents, or any other light-emitting source to effectuate anynumber of light-mitting effects from a LEPD (described below).

Additionally, a system may be provided for processing at least one of anaudio or video input for non-scripted light modulation of at least onelight-emitting peripheral device (LEPD). The system comprises at leastone LEPD in at least one of physical contact or free from at least oneuser and in communication with at least a first device (D1) playingprogramming; a processor; a memory element coupled to the processor; aprogram executable by the processor to: recognize at least one of theaudio or video input from the D1 and determine for at least one taggedevent, at least one of a pixel color score, a pixel velocity score, anevent proximity score or an audio score; and convert the at least onescored event into at least one of an output command that triggers orcontrols a light-emitting effect of the at least one LEPD, whereby theLEPD is further at least one of a mouse, keyboard, headset, speaker,joystick, D1-coupled display, television monitor, tablet, smart phone,room light source, or home IoT (Internet-of-Things) hub, and whereby theDI is at least one of a gaming console, desktop, laptop, tablet,smartphone, playback device, television monitor, display, or home IoThub.

In yet other system embodiments, a system may be provided for processingat least one of an audio or video input for non-scripted lightmodulation of at least one light-emitting peripheral device (LEPD),wherein the at least one LEPD in at least one of physical contact orfree from at least one user and in communication with at least a firstdevice (D1) playing programming; a processor; a memory element coupledto the processor; a program executable by the processor to: recognize atleast one of the audio or video input from the D1 and determine for atleast one tagged event, at least one of a pixel color score, a pixelvelocity score, an event proximity score or an audio score; and commandat least one of a trigger or control over a light-emitting effect of theat least one LEPD upon a threshold-grade score.

In some embodiments, the at least one LEPD may be a device forcontrolling operation of at least one of an original programming feed orlive feed displayed on the screen coupled to the at least one D1, suchas keyboard, mouse, tablet, or joystick. The at least one D1 and the atleast one LEPD may be the same device. In some configurations, aplurality of LEPD's may be in communication to the at least one D1, suchas an array of lighting sources/fixtures. Further, the arrayedconfiguration may comprise a plurality of LEPD's with at least one inphysical contact with the at least one user and in communication to theat least one D1. For instance, the array of lights may emit light insynchrony or in concert with a light-configured keyboard or joystick fora heightened sensory experience. In yet other configurations, aplurality of LEPD's with at least one free from the at least one userand in communication to the at least one D1 may be possible. Forinstance, a headset and keyboard may emit light in synchrony or inconcert with a light source/fixture or an array of lightsources/fixtures to achieve the heightened sensory experience.

In some embodiments, the at least one D1 may further comprise at leastone of a camera, microphone, sensor, or audio or video capture forplaying a live feed and the at least one LEPD in physical contact withthe user is at least one of a mobile phone, biomedical tool, erotic toy,steering wheel, or automobile pedal. In yet other embodiments, the atleast one D1 is at least one of a camera, microphone, sensor, or audioor video capture for playing a live feed and the at least one LEPD isfree from the user and is at least one of an automobile alarm beacon,house alarm beacon, stand-alone haptic tower, heat lamp, light source,light fixture, thermostat, or IoT hub.

In some embodiments, a processor may further be configured with specificmodules to perform the recognition of the input for the tagged event,and then trigger or control the light-emitting effect of the at leastone LEPD based on the scored tagged event being above a pre-definedthreshold. In an embodiment (not shown), the processor may comprise ofat least one of an a/v recognition block (recognition block), aconversion block, and a controller (modulator). The recognition block(engine) determines a proximity score of the tagged event by determiningthe distance from any one of, a selected target zone comprising theevent and, or a selected destination zone comprising the user, within amatrix of zones that occupy the entire field of view and, or sound. Therecognition block may further determine a pixel color score of thetagged event by calculating an average hue score of the tagged eventusing pixel data in a screen buffer by calculating a nearnesscoefficient, calculating an average of color channels in the screenbuffer, calculating an offset coefficient, calculating an averageluminance in the screen buffer and deriving the average pixel score ofthe tagged event based on an aggregation of the coefficients. Pixelvelocity scores of the tagged event are calculated based on thecoefficient by capturing a series of frames, and calculating acoefficient related to pixel velocity by testing the per-frame andper-range delta in any one of, or combination of, hue, luminance,brightness, saturation and, or color value. Further yet, an audio scoreof the tagged event may be determined based on a coefficient bycapturing an audio buffer and calculating an Average Energy, ImmediateEnergy, Immediate Energy Delta & Immediate Energy Mean Deviation andfurther, calculating a coefficient related to broad and narrow changesin a frequency spectrum.

Further yet, in an embodiment of the invention, the conversion block(engine) may apply a scoring rule for the conversion of at least onethreshold-grade scored event into a light-emitting output command.Further yet, the conversion block is further coupled to a controller(modulator), which further, processes the light-emitting output commandfor actuating at least one of a trigger or control of a light emittingeffect of any peripheral-whether it be in physical contact or free theat least one user. Furthermore, the processor/recognitionblock/conversion block/controller/modulator/engine may comprise afeed-forward and, or back-propagated neural network trained to triggeran output based on any one of, or combination of, a stored data input,stored tagged event, stored coefficient value, stored event proximityscore value, stored pixel color score value, stored pixel velocity scorevalue, stored audio score value, and, or stored output command. In someembodiments, the recognition block may tag at least one event forscoring by at least one of, motion, shape, color or sound. In yet otherembodiments, the recognition block may score at least one event for atleast one of a pixel color score, pixel velocity score, pixel proximityscore, or audio score.

The following is an exemplary list of displayed events that can berecognized to trigger lighting effects (LEDs) on any host of LEPD's toaugment the experience:

Displayed Events:

-   -   1. Fire    -   2. MotionFlow—visualizing motion through space: flying, sky        diving, walking, running, etc.    -   3. Lightening    -   4. Looking down gun barrels    -   5. Being underwater    -   6. Getting health or using a health pack    -   7. Getting into a vehicle    -   8. Recognizing a vehicle    -   9. Recognizing a flying vehicle    -   10. Recognizing a motorcycle    -   11. Recognizing objects—frying pan (Pubg), camp fire (fortnite),        Bouncy launcher (fortnite)    -   12. Taking damage    -   13. Special attack indicator    -   14. Looking at menus    -   15. Looking at inventory

Specific Displayed Events/Elements:

-   -   1. Apex Legends/Razer Chroma        -   a. Opening Apex packs and rewards        -   b. Notifications as you fire, heal and take damage        -   c. Alerts when your ultimate is ready        -   d. Smoke trails when skydiving (Apex Legends)        -   e. Pick up loot of different rarity (i.e. white, blue,            purple, gold)    -   2. Assassin's Creed/MSI Ambient Link        -   a. Blocking an attack        -   b. Skill effects        -   c. Health skill use        -   d. In game events        -   e. Looting stuff        -   f. Getting hit, etc.    -   3. Call of Duty: Black Ops 4/Asus Aura        -   a. Under water elements detected with blue keyboard        -   b. Timer count down on keyboard LED's        -   c. Meters for life, etc., indicated by light on any keyboard            keys        -   d. Blackout circle flashes keyboard white        -   e. Player death results in red keyboard

Light-Emitting Effects Driven:

-   -   1. Output a nova on the keyboard    -   2. Output a sparkle on the keyboard    -   3. Raining effect    -   4. Outward “spiral” effect    -   5. Wave effect    -   6. Inward collapsing effect    -   7. Rainbow effect    -   8. Emojis (smiley face lit on the keyboard upon a victory; poop        emoji lit upon a defeat)    -   9. Slashing sword effect across the keyboard    -   10. Strip sliding across the keyboard

Now in reference to FIG. 17—an exemplary process flow diagramillustrating the customizable and immersive effect delivery inaccordance with an aspect of the invention. In one embodiment, theend-user of the customizable immersive effect from a rendered web-pagebegins by entering a script input from any one of a basic scriptlanguage (HTML, HTML5, Javascript (JS), or any graphics element usingJavascript, such as Canvas) for custom-tuning effects from an endperipheral device (EEPD/LEPD) 1702. Any one of a low-coding language,other than HTML or JS, may be used as well, provided the renderingmodule is configured with an interpreter for it. This end-user script(EUS) input may be performed on a layer imported. loaded or exportedonto a browser page. The end-user may enter any one of theaforementioned script—each with a low-barrier of coding-allowing for theend-user to ad-hoc create an effect, ad-hoc create an interface switchor control for an effect, or post-hoc refine/augment an existinginterface switch or control 1706. Certain meta-information in the EUSmay command for this. For example, an end-user may input a meta-tag tocreate a slider or slide bar as a color gradient for choosing thespecific color to be rippled onto the end peripheral-enhancing/tuningthe already existing limited text-based listing of colors. Thisswitch/control creation or tuning may be performed prior to or after therendering module renders the web-browser page-post EUS.

In one embodiment, a processor (rendering module) is configured tointerpret the EUS 1704 and render the web-browser page or web-page intoat least a two-dimensional effects plane 1707 subject to ageo-positional transform/scaling of virtual EEPD/LEPD and screen regiongrabbing 1708 by a transformer/grabber module. Any one of these modules,interpreters, compilers, etc. may be any one of, or combination of,software, firmware, or hardware, performing at least one of parsing orexecuting stored pre-compiled machine-independent code, linked atrun-time, resulting in the rendered effects plane or stage-subject fortransforming and screen region grabbing—for translating the virtualeffect/device into the physical environment surrounding the end-user.Traditional methods are heavy and typically need to be compiled. Forexample, C/C++ logic is converted to machine code and carries all theweight and complexity of the target application with it. Interpretedlanguages do not require ‘compilation’, but rather, are loaded on thefly. While performance may be effected when ‘loaded on the fly’, allsignal processing and analytics may be performed in C/C++, and thenexposed to a browser (where rendering has been the primary focus for along time, and the technology is fast for drawing), thus availing of thebest of both worlds. Furthermore, there isn't a need to install aninterpreter or virtual machine (as you would with python, java, or someother interpreted languages).

In one embodiment, a method for controlling at least one light-emittingperipheral device (LEPD) for an immersive light effect comprises thesteps of: providing a web-browser page interface configured for scriptinput (EUS) 1702 for adjusting any one of an aspect of the immersivelight effect 1706 from the at least one LEPD; rendering thescript-inputted web-browser page to an off-screen buffer 1704 visualizedas at least a two-dimensional effects plane 1707; applying ageo-positional transform and scaling of virtual LEPD's within theeffects plane and capturing at least a region of the rendered webpage1708; and controlling a light effect emitted from the at least one LEPD1709 corresponding to the effects plane transformed/scaled virtual LEPDand captured region of the rendered web-browser page.

As shown in FIG. 17, the end peripheral devices 1706 may include for anyeffect-emitting peripheral device that may be geo-positionally trackedand scaled on the at least two-dimensional effects plane for physicalsurrounding correspondence. This may include for LEPD's, such aslight-emitting (LED) strips, keyboards, bulbs, along with EEPD's, suchas haptic vests, and heated air-dispensing devices. The interface forEUS (optionally, script layer) may be an application running on aprocessor or microcontroller and may be imported, exported, orpre-loaded onto the web-browser page.

The script layer 1804 may appear as a split-screen with thedirected/imported/loaded web-browser page 1802 (see FIG. 18—screen shotillustrating an exemplary script layer on a web-browser page fordelivery of the customizable and immersive effects in accordance with anaspect of the invention). While somewhat difficult to ascertain fromFIG. 18, shown to illustrate the point, an end-user may load the scriptlayer “Hello World.html” which contains the text “Hello World” 1804. Theend-user would then load a local html file that redirects to thehomepage for Whirlwind VR for content 1802 displayed adjacent to thescript layer. For all intents and purposes, the method is rendering awebpage that may orchestrate or ripple any number of immersive effectson any type of end peripheral (LED strips, keyboards, game-playcontrollers, etc.) based on the transform/scale/content within theregion grab. The web browser page may be any processing block renderingwebsites by processing script input and converting it to user-facingcontent on a local client. The script layer 1804 is plugging in customfeatures to this ‘web browser’ for handling controllable parts of theeffect. For example, if you add a special tag to the html effect itautomatically ripples into the application UI and is exposed to theend-user as an option-either de novo or an enhanced add-on. For example,if the end-user inputs the following EUS:

<meta property=“speed” label=“Rainbow Speed” type=“number” min=“1”max=“10” default=“0”>

The above EUS creates a slider control in the UI for the end-user toachieve a desired effect. Additionally, a corresponding object inJavascriptCore called ‘speed’ may be created, and may be subsequentlyqueried it to ascertain how fast the effect runs. Other controls thatmay be scripted for in the UI may be drags, drops, clicks, dials, colorpickers, color wheels, switches, coordinates, text fields, or any visualdepiction of effect choices to select from. The visual depictions ofeffect choices may be adjustable for at least one of a color, hue,effect, effect speed, threshold-based, or audio-based. The advantagewith the EUS and rendering of a scripted webpage is that the end-usercan create customized complex effects in their physical surrounding, allwhile created from a single file. Furthermore, the controlsoftware/system and ensuing methods does not need to be recompiled torender the effect. Lastly, the system/method can load the desired effectfrom a remote URL if desired, with no local script required. In oneembodiment of this, a library of effects may be stored, previewed, andloaded, without the need to EUS de novo.

While not shown in FIG. 17 or 18, in other embodiments, an EUS layer orEUS is not required, and just content from the grabbed rendered web-pageand the basic UI effects controls without scripted customization mayripple the desired effects. The content may be static (pre-scripted witheffect triggers), dynamic (not pre-scripted with effect triggers, ratherrelying on a computer vision-CV-analysis of at least one of pixel colorscore, pixel proximity score, pixel velocity score, pixel motion, zonecolor, zone motion, audio score, audio intensity score, audio spectraldensity, etc.), or inert (neither pre-scripted with triggers, norrelying on CV techniques to determine triggers). In yet otherembodiments, EUS and content from a grabbed rendered webpage may not berequired, and simply grabbed rendered webpage of standard effects.

Now in reference to FIG. 19, which illustrates a block diagram of anexemplary system in accordance with an aspect of the invention. Asshown, the system comprises a processor (effects engine 1902) furthercomprising a rendering module 1902 a, an effects plane 1902 b, atransformer/grabber 1902 c. The modules interact to achieve a specificroutine/sub-routine for controlling at least one effect-emittingperipheral device (EEPD) for a customizable and immersive effect. Theeffects engine layers at least one of a static, inert or dynamic contentfrom a web-browser page with at least one of an end-user script (ESU)input for custom effects or standard effect requests via a standarduser-interface input. The rendering module 1902 a renders the scriptedweb-browser page to an off-screen buffer visualized as at least atwo-dimensional effects plane 1902 b. In one embodiment, thetransformer/grabber 1902 c enables the end-user to apply geo-positionaltransform and scaling of virtual LEPD's within the effects plane andcapture or grab at least a region of the rendered webpage. In anembodiment, a customized EUS effect is emitted from the at least oneEEPD or end peripheral 1904 corresponding to the captured/grabbed regionof the web-browser page featuring an effect or content and thecorresponding effects plane 1902 b transformed/scaled.

For example, consider a scenario of an erupting volcano within a grabbedregion of the rendered web-browser page, wherein the grabbed content isEUS/transformed/scaled/CV-recognized by the system to ripple a gradientof red-orange light in an irregularly cascading fashion starting from atop right corner of a back-display adhered LED strip down to the rightportion of the end-users keyboard and finally trailing down to an LEDstrip adhered to a wall-floor mold behind the user. Based on thefollowing EUS for a cascading effect to correspond to volcaniceruption/lava flow; and geo-positioning/scaling of LED strips, keyboard,and heated air-dispensing device for maximal dramatic effect for devicesor portions of devices positioned on top, to the right, and behind theuser, the visual scene of a volcanic eruption and lava flow within thegrabbed region will be CV analyzed to perceive heat, eruption sound,flowing sound, rumbling sound, crackling sound, eruption color, lavaflow color, wind velocity, burning sensation, flying debris, and suddenimpact, etc. to ripple the rendered effect from the real-world endperipheral devices

Furthermore, as the red-orange light cascades from top to bottom in anirregular fashion-along the right side of the user, the heated-airdispensing device positioned on the right of the user will activate asthe virtual volcanic eruption is closest to the virtual user orcharacter from the viewing experience. Further yet, if the volcanoeruption is under control and, or if the user/character moves fartheraway from the eruption site, then the effects engine will counter-effectby restoring the base-line (pre volcanic eruption) conditions (less warmair dispensed, fainter eruption, crackling, or lava flowing sounds, anda restoration of the ambient lighting).

Now in reference to FIG. 20—illustrating an exemplary process flow ofthe effects engine in accordance with an aspect of the invention. Theeffects engine/rendering module renders combinatorial/layered effectsmirrored onto end-devices (light-emitting devices, inter alia) based onhybrid triggers (combination of different triggers in any sequenceand/or concurrence) embedded/coded/scripted/computer-vision (cv)activated from an audio/video game/video output. As shown in FIG. 20,the system comprises a rendering module and a transform/scale module,and optionally, a peripheral device controller or control system.Collectively, the system or method provides for coordinated delivery ofeffects onto an end peripheral device based on a captured region andtransformed/scaled application 2007. The end peripheral device, oreffects-emitting peripheral device (EEPD), may encompass the LEPD(RGB-keyboard, mouse, head-set, PC/console tower, PC/console towercomponents, light-diode strip, bulb, etc.), along with any one of ahaptic device, such as a controlled air-dispensing device, haptic vest,etc. 2008.

As a result, an end-user, with just minimal coding/scripting (html,Java) experience, may script customized and complex effects 2003—layeredwith other triggers such as developer-coded triggers for specificelements/events/alpha-numeric characters 2004, user-adjusted/dialed/slidbasic effects 2006, or computer vision techniques (pixel analysis)2001—driving combinatorial/layered effects from EEPD's 2009 mirroring agrabbed region of the rendered, transformed, and scaled display of thedigital canvas-positioned virtual device 2005 of the physical EEPD 2009.The end result of the combinatorial/layered effects from this hybridtrigger approach (Hy-Fx) is the countering of false positives andlatency of effects from an end-device during a viewing/gamingexperience-both of which severely restrain a user from experiencingtotal immersion. To this end, the Hy-Fx solution additionally offersoptions for delivery of complex combinatorial/layered effects that maybe rippled from end-devices mirroring/corresponding to keyelements/events from the engaged content.

FIGS. 21 and 22 illustrate a block diagram of an exemplary system inaccordance with an aspect of the invention. In a preferred embodiment,the Hy-Fx system comprises at least one end-device (E-D) or peripheraldevice in communication with at least a first device (D1) outputtingaudio/video programming; a processor; a memory element coupled to theprocessor; a program executable by the processor to: position a virtualrepresentation of the E-D on a digital canvas displayed on a D1-coupleddisplay representing a user's physical and virtual space; and capturefrom a corresponding region of the virtual space for modulating aneffect 2108 on a corresponding portion of the E-D based on a combinationof at least two different triggers 2102 a/2202 recognizing the a/velement (health meter icon/alpha-numeric (circle, needle, bar, range,etc.), score, kill meter icon/alpha-numeric, ammunition level icon,etc.).

Alternatively, the capturing from a corresponding region of the virtualspace for modulating an effect 2108 on a corresponding portion of theE-D may be based on a combination of different triggers 2102 a/2202recognizing a/v events in the gaming/viewing experience (volcanicexplosion, weapon firing, shots taken, etc.).

In other embodiments, the capturing from a corresponding region of thevirtual space for modulating an effect 2108 on a corresponding portionof the E-D may be based on a combination of different triggers 2102a/2202 recognizing a/v events and a/v elements in the gaming/viewingexperience (health meter icon/alpha-numeric, score, kill metericon/alpha-numeric, ammunition level icon volcanic explosion, weaponfiring, shots taken, etc.). In yet other embodiments, the capturing froma corresponding region of the virtual space for modulating an effect2108 on a corresponding portion of the E-D may be based on a singletrigger 2102 a/2202 recognizing either an a/v event or an a/v element.It should be appreciated by a person of ordinary skill in the art thatany sequence or temporal scheme of a/v events and/or elements triggeringthe combinatorial/layered effects (light modulation) 2108 from the E-D(RGB-keyboard, mouse, console/PC tower) may be possible by the Hy-Fxsystem/engine. To reiterate, the combinatorial/layered effects resultingfrom the single, combination, or combination of different a/velement—a/v event triggers from the Hy-Fx engine delivers a versatilearray of complex, multi-dimensional effects (dynamic lighting, etc.) onany one of an E-D (RGB-keyboard, etc.) to further enhance the immersiveexperience during content engagement (role-play video gaming, etc.).

As depicted in FIG. 21/22, the effects engine/rendering module2102/trigger library 2102 a/2202 may comprise at least one of thefollowing a/v-element/event recognizing triggers: optical characterrecognition (alpha-numeric characters-meters, gauges, scores, shields,warnings, informationals, etc); end-user scripted triggers (supra, para.175-183); developer scripted triggers (embedded cues and/or graphicaladjusters); computer vision triggers (analysis of real-time events basedon at least one of pixel color score, pixel velocity score, pixelproximity score, or an audio score, supra, para. 166-174).

Now in reference to FIGS. 23 and 24—illustrating an exemplary methodflow of the Hy-Fx engine. As shown in FIG. 23, the method for processingan audio/video element and/or event from an audio/video input forend-device modulation may be based on a trigger, triggers, and/ordifferent triggers comprising the step of: (1) positioning a virtualrepresentation of the end-device on a digital canvas displayed on aD1-coupled display representing a user's physical and virtual space2302; and (2) capturing at least one a/v element and/or event from acorresponding region of the virtual space for modulating an effect on acorresponding portion of the end-device based on at least one of asingle or combination of different triggers recognizing the a/velement-a/v event 2304.

In reference to FIG. 24, which depicts the method for processing anaudio/video element from an audio/video input for light modulation of anRGB-keyboard based on at least one of a single, combination of triggers,or combination of different triggers. The method may comprise the stepsof: (1) positioning a virtual representation of the RGB-keyboard on adigital canvas displayed on a D1-coupled display representing a user'sphysical and virtual space 2402; and (2) capturing at least one a/velement from a corresponding region of the virtual space for modulatinga lighting effect on a corresponding portion of the RGB-keyboard basedon at least one of a single, combination of triggers, or combination ofdifferent triggers recognizing at least one of an a/v element or a/vevent.

FIG. 25-31 represent screen-shots of exemplary sequence of events duringa representative game-play. FIG. 25 illustrates a user being prompted togeo-position his desired end-device into the virtual space. As shown,the white rectangle 2502 (corresponding to a RGB-keyboard for the user)is positioned in the lower center of the digital canvas since theRGB-keyboard in the users physical space is positioned in asubstantially lower center area in relation to the display. In the eventthat the right-handed user also wants to Hy-Fx activate his mouse, hewould canvas-position a small square corresponding to the mouse on theright of the canvas-positioned RGB-K, and vice-versa if left-handed,rippling effects onto the RGB-K and mouse mirrored from thecaptured/grabbed digital canvas/screen/display.

FIG. 26 represents a screen-shot of an exemplary user-end script layerfor triggering end-device effects (end-user scripted effects). Theend-user may load a local html file that redirects to the homepage forWhirlwind VR for content 1802 displayed adjacent to the script layer.For all intents and purposes, the method is rendering a webpage that mayorchestrate or ripple any number of immersive effects on any type of endperipheral (LED strips, keyboards, game-play controllers, etc.) based onthe transform/scale/content within the region grab. The web browser pagemay be any processing block rendering websites by processing scriptinput and converting it to user-facing content on a local client. Thescript layer is plugging in custom features to this ‘web browser’ forhandling controllable parts of the effect. For example, if you add aspecial tag to the html effect, it automatically ripples into theapplication UI and is exposed to the end-user as an option-either denovo or an enhanced add-on. For example, if the end-user inputs thefollowing EUS:

<meta property=“speed” label=“Rainbow Speed” type=“number” min=“1”max=“10” default=“0”>

The above EUS creates a slider control in the UI for the end-user toachieve a desired effect. Additionally, a corresponding object inJavascriptCore called ‘speed’ may be created, and may be subsequentlyqueried to ascertain how fast the effect runs. Other controls that maybe scripted for in the UI may be drags, drops, clicks, dials, colorpickers, color wheels, switches, coordinates, text fields, or any visualdepiction of effect choices to select from. The visual depictions ofeffect choices may be adjustable for at least one of a color, hue,effect, effect speed, threshold-based, or audio-based. The advantagewith the EUS and rendering of a scripted webpage is that the end-usercan create customized complex effects (with minimal coding experience)in their physical surrounding, all while created from a single file.Furthermore, the control software/system and ensuing methods does notneed to be recompiled to render the effect. Lastly, the system/methodcan load the desired effect from a remote URL if desired, with no localscript required. In one embodiment of this, a library of effects may bestored, previewed, and loaded, without the need to EUS de novo.

FIG. 27 illustrates a screen-shot of an exemplary scene from the popularvideo game “Fortnite” (Epic Games). As shown, starting from the topcenter and working clockwise around the edge of the scene, each red boxcontains vital information related to the user/player. For example, thetop is the compass or direction the player is facing. The next box belowthat is the “First Strike” pop-up that informs the player that he/shegot the first kill of the match. The four boxes in the top right cornerare the time until the circle perimeter closes; the number of playersleft in the match; the number of kills that the player has; and thenumber of kills the team has. The next three boxes are resources. Thebottom left corner is the player's level and above that is the healthand shield meters. Once a light-script developer has identified whichelement he/she wants to capture and what he/she wants to happen whencertain thresholds are met, the developer thinks about what theexperience should look like. For instance, the developer may identify anumber change in a top-right corner, which typically registers/displaysupon a “kill”. Similarly, the developer could look for the letters/word“First Strike” and that could trigger a special golden wave effect. Byusing this Optical Character Recognition (OCR) trigger look-up, thelight-script developer will identify the coordinates of the letters,words, meter on the screen and then use OCR to look for “First Strike”in those predictable locations.

Upon identifying, the developer may program the special golden waveeffect in an HTML/Javascript as shown in FIG. 28 (illustration of ascreen-shot of an exemplary HTML/Javascript document in accordance withan aspect of the invention). In continuing reference to FIG. 28, metatags are evident assigning “meter/health” with a predictable position onthe screen. Optionally, tags may be assigned to symbols/elements/iconsrepresenting “meter/health”, or any other vital player-related info.Also evident is an HSL value or color value that we have to scrape oridentify from the game in real-time (computer vision trigger). Once thevalues are determined, the light-script developer can code coloredeffects that ripple or occur on the canvas as shown in FIG. 29(Illustrating a screen-shot of an exemplary digital canvas projection ofRGB-keyboard light emission in accordance with an aspect of theinvention). Those effects can be sweeps in any direction and theperipherals or end-devices automatically capture the region set on thecanvas for rippling or emitting the effect-providing enhanced immersionto the player. The following list represent typical “look-up” triggersthat may be combined for additional complexity and accuracy:

-   -   a. Window title.    -   b. OCR=Optical Character Recognition=reading words, letters,        numbers in real time.    -   c. Meter recognition=linear meters, circle meters, arc meters,        segmented meters, etc.    -   d. Icon recognition=placards of a static image or indicator.    -   e. Audio recognition=specific sounds like footsteps, explosions,        or any kind of specific audio cue.

The Hy-Fx approach may combine any number of triggers (developer-end“look-ups”, end-user scripted, CV, etc.), in addition to the combinationof the above “look-up” triggers to create the layered/complexity ofeffects. Additionally, the Hy-Fx approach (using a layered/combinatorialtrigger approach) ensures against false positives that may normallyoccur under a single trigger scheme. For instance, a player roamingthrough a grassy field, whereby the grassy field appears on a portion ofthe frame co-located with a predicted frame position of a typicallygreen health shield/meter, should not trigger an effect coded for aheath meter since the player has not changed his health meter by takinga hit or improving his health status. In addition to ensuring againstfalse positives, the Hy-Fx approach, with the combination of triggers,adds complexity to the effects by layering effect onto another. (FIGS.30 and 31 depict exemplary RGB keyboards rippling thecomplex/layered/accurate effects mirrored from the key/targeted a/vevents and/or a/v elements).

Therefore, the foregoing is considered as illustrative only of theprinciples of the invention. Further, since numerous modifications andchanges will readily be apparent to those skilled in the art, allsuitable modifications and equivalents may be considered as fallingwithin the scope of the invention.

1. A system for end-device modulation by a hybrid trigger, said systemcomprising: At least one end-device (E-D) in communication with at leasta first device (D1) outputting audio/video (a/v) programming; aprocessor; a memory element coupled to the processor; a programexecutable by the processor to: position a virtual representation of theE-D on a digital canvas displayed on a D1-coupled display representing auser's physical and virtual space; and capture from a correspondingregion of the virtual space for modulating an effect on a correspondingportion of the canvas-positioned E-D based on a combination of at leasttwo different triggers recognizing at least one of an a/v element or a/vevent from the a/v program.
 2. The system of claim 1, wherein the E-D isa vibro-tactile or haptic device.
 3. The system of claim 1, wherein theE-D is a light-emitting peripheral device (LEPD).
 4. The system of claim3, wherein the LEPD is a RGB-Keyboard.
 5. The system of claim 3, whereinthe LEPD is a light-emitting gaming device comprising at least one of adisplay/monitor, mouse, game controller, head-set, computing tower, gameconsole, any portion and/or component of a computing tower, light-strip,bulb.
 6. The system of claim 4, wherein the RGB-keyboard lightingeffects triggered by at least one of the a/v element or a/v event fromthe program is at least one of a nova, sparkle, rain, spiral, wave,collapse, rainbow, emoji, slash, sliding strip, cascade.
 7. The systemof claim 1, wherein the a/v element recognized is at least one ofopening packs/rewards; notifications as you fire, s trike, blockstrikes, take damage, heal, death; health alerts; health meters; timers.8. The system of claim 1, wherein the trigger is a computer visionanalysis of an a/v event comprising at least one of a pixel color score,pixel velocity score, pixel proximity score, or audio score.
 9. Thesystem of claim 1, wherein the a/v event recognized is at least onefire, explosion, lava, lightening, freezing, taking a hit, gaininghealth, flying, sky diving, walking, running, falling, jumping, lookingdown a gun barrel, being underwater, getting into a vehicle, a flyingvehicle, objects—frying pan, camp fire, launcher, special attackindicators, menus, inventory.
 10. The system of claim 1, wherein thetrigger is an audio score.
 11. The system of claim 1, wherein thetrigger is a collection of alphabet and/or numerical recognized by anOptical Character Recognition (OCR).
 12. The system of claim 1, whereinthe trigger is a user-end-scripted trigger corresponding to at least oneof an a/v element or a/v event from the program.
 13. A system forprocessing an audio-video element in an audio-video output for lightmodulation of an RGB-keyboard, said system comprising: the RGB-keyboardin communication with at least a first device (D1) outputting videoprogramming; a processor; a memory element coupled to the processor; aprogram executable by the processor to: position a virtualrepresentation of the RGB-keyboard on a digital canvas displayed on aD1-coupled display representing a user's physical and virtual space; andcapture at least one a/v element from a corresponding region of thevirtual space for modulating a lighting effect on a correspondingportion of the RGB-keyboard based on a combination of triggersrecognizing the a/v element.
 14. The system of claim 13, wherein theRGB-keyboard lighting effects are at least one of a nova, sparkle, rain,spiral, wave, collapse, rainbow, emoji, slash, sliding strip, cascade.15. The system of claim 13, wherein the a/v element recognized is atleast one of opening packs/rewards; notifications as you fire, s trike,block strikes, take damage, heal, death; health alerts; health meters;timers.
 16. The system of claim 13, wherein the trigger is a collectionof alphabet and/or numerical recognized by an Optical CharacterRecognition (OCR).
 17. A system for end-device modulation by a trigger,said system comprising: at least one end-device (E-D) in communicationwith at least a first device (D1) outputting audio/video programming; aprocessor; a memory element coupled to the processor; a programexecutable by the processor to: position a virtual representation of theE-D on a digital canvas displayed on a D1-coupled display representing auser's physical and virtual space; and capture from a correspondingregion of the virtual space for modulating an effect on a correspondingportion of the canvas-positioned E-D based on a combination of at leastone triggering a/v event.
 18. The system of claim 17, wherein the E-D isa vibro-tactile or haptic device.
 19. The system of claim 17, whereinthe E-D is a light-emitting peripheral device (LEPD).
 20. The system ofclaim 19, wherein the LEPD is a RGB-Keyboard.
 21. The system of claim19, wherein the LEPD is a light-emitting gaming device comprising atleast one of a display/monitor, mouse, game controller, head-set,computing tower, game console, any portion and/or component of acomputing tower, light-strip, bulb.
 22. The system of claim 20, whereinthe RGB-keyboard lighting effects triggered by the a/v event from theprogram is at least one of a nova, sparkle, rain, spiral, wave,collapse, rainbow, emoji, slash, sliding strip, cascade.
 23. The systemof claim 17, wherein the trigger is a computer vision analysis of an a/vevent comprising at least one of a pixel color score, pixel velocityscore, pixel proximity score, or audio score.
 24. The system of claim17, wherein the a/v event recognized is at least one of a fire,explosion, lava, lightening, freezing, taking a hit, gaining health,flying, sky diving, walking, running, falling, jumping, looking down agun barrel, being underwater, getting into a vehicle, a flying vehicle,objects—frying pan, camp fire, launcher, special attack indicators,menus, inventory.
 25. The system of claim 17, wherein the trigger is anaudio score of a certain threshold or range.